Pantek Library
Hosting Provided By
CybrHost
High Speed Hosting
Mailing Lists: lists.debian.org > debian-boot > 07 > 060176.html (Request Expert lists.debian.org Support)

[long] Management of the installation guide - part I

From: Frans Pop <elendil(at)planet.nl>
Date: Sat Jun 30 2007 - 13:36:43 EDT


(CCing the list as I think it is useful to have this archived and I guess it could be interesting to others as well)

Hi Felipe,

I hope you do not yet regret your impulsive reaction of agreeing to take over management of the installation guide ;-) Anyway, thanks a lot!

I suggest we take this step-by-step. First I think would be to get you started taking care of the "daily" builds; we can discuss other areas like later; these include:
- troubleshooting build problems

  • official builds for uploads
  • official builds for the website
  • inclusion of the manual on CD images
  • release management
  • acting as editor (optionally)

If you'd like to use my mails and things you learn along the way to create some sort of more structured and permanent documentation, that would be fine by me. I think it would be very useful.

I put "daily" in quotes because I have never actually done them daily. There really is no need for that as the mutation frequency is not all that high. When I had the builds cronned, they were done 4 times per week; currently I run them manually [1] and basically as needed. IMO it would be good to have the builds running more regular again. It is also good to run more frequent builds, for example when a release is scheduled and loads of translation updates are being committed.

Types of builds

There are basically four types of builds:
- single, manual builds for a specific architecture/language
  uses build/buildone.sh from the commandline
- daily builds for alioth using the infrastructure described below

  • official builds for uploads done from debian/rules which calls build/build.sh
  • official builds for the website done on www-master using build/buildweb.sh; triggered by uploads
Do you need help?X

Setting up the build environment

The following commands should be enough to set up a build environment: $ cd <wherever you want to put this>

This directory will be $ROOT. I'd suggest /srv/d-i_manual.daily/. Make sure you are allowed to write to this directory and that correct permissions are set for files and subdirs that are created in it. 

$ mkdir bin
$ mkdir manual
$ cd bin
$ svn co svn+ssh://svn.debian.org/svn/d-i/trunk/scripts/manual-daily

This first checkout gets you the daily build scripts. It is fine to make changes here and commit these.
I'll introduce the files in this directory later. 

$ cd ../manual
$ svn co -r 47661 svn+ssh://svn.debian.org/svn/d-i/trunk/manual
$ dpkg-checkbuilddeps
$ sudo aptitude install 
$ sudo aptitude install rsync openssh-client # if not already there

I have included '-r 47661' because that is the revision for which I have run my last daily build, so that is where you should start. If you don't have time to set up things quickly enough, I can continue to run builds, but we should then agree on a new revision for you to start at.

You need a separate checkout of the manual the daily builds because builds are triggered by updates (an 'svn up' is done at the beginning of a build). This also means that as a general rule you should never trace or fix build problems in this checkout, but rather use a "normal checkout" to trace, fix and test the problem, then commit. After that just run a daily build again (possibly a retry) and let that pull in the changes.

Next, you'll need to create a config file in the $HOME of the user you will use to run the builds; this should probably just be your own account as you need commit and upload access to alioth/svn. This config file is sourced by several of the build scripts. I created it for ensure consistency and to make sure I only needed to make changes in one place instead of having to grep endlessly and still forget changes in certain places.

For me, this file contains:
<snip>
$ cat ~/.d-i_manual_daily
# Configuration settings for "daily" builds of the Installation Guide

Do you need more help?X

ROOT=/srv/d-i_manual.daily
source=$ROOT/manual
destination=$ROOT/build
logdir=$ROOT/log

NAME_FROM="Frans Pop"
MAIL_FROM="elendil@planet.nl"
</snip>

The first set of values tells the scripts where to find/write things. The second set is used to send log mails for the builds (please change the mail address and name!). So you'll also need to make sure that the system is correctly set up to send mail.
If you intend to run the builds cronned, make sure that you also receive any mails cron will send if there is output from cronjobs.

Note that the scripts currently assume that you can just 'svn ci', 'scp' and 'rsync' to alioth using an ssh key; if you need anything special, you'll probably need to modify things.

Contents of the build environment

Directories: 

./bin    - scripts and support files; from D_I SVN ./scripts
./manual - local checkout of the manual for daily builds
./build  - place were built files are moved; rsynced to alioth from there
./log    - log files are created here; some are copied to alioth

Files in ./bin:
README                - ...
archlist              - architectures to be built + descriptions
langlist              - languages to be built + various info needed
build-main            - build wrapper script: this is what you execute
                        (indentation in following lines indicates calls)
.build-manual         - main build script: does the actual builds and
                        copying of results to alioth
..create-index        - creates index page for dev. version of manual [2]
...translation-stats  - calculates translation stats for XML based transl.
...po_stats/          - script/files to calculate translation stats for
                        PO based transl.
.mail-logs            - sends log mails after build completes/fails
index.head            - template file for index page used by create-index
index.single          - same (table of langs for which only i386 is built)
index.stats           - same ("statistics" section)
index.pohead          - same ("PO statistics" section)
index.tail            - same (page footer)
index.changes         - items for "Recent changes" section
translators.html      - web page linked from index page

Can we help you?X

msgtxt.common         - template for build log mails to translators
msgtxt.ok             - same (for successful builds)
msgtxt.fail           - same (for failed builds)
nopdf.pdf             - placeholder file uploaded if build of PDF fails
notxt.txt             - placeholder file uploaded if build of text fails

Running the builds

You will need to study the scripts themselves for the gory details about how things are done. Most scripts do have some comments to explain what's happening and why. Running with 'set -x' will probably also help a lot to understand the details :-)

Can't find what you're looking for?X

Normally you should just execute the 'build-main' script. If I run a build manually I will mostly do:
$ ./build-main &
$ tail -f ../log/main.log
The tail of the overview log allows me to follow the build.

You can also execute 'build-manual' separately if you want to avoid sending log mails for some reason. It is possible to manually run 'mail-logs' afterwards; that can also be run separately to resend the log mails from the last build run.

The steps for a build run are follows:
- check for lock files (see next section)

  • create lock file to prevent two builds running at the same time
  • do 'svn up'
  • if there are updates for en:
    • build en version
    • if successful, move to output directory
    • update POT files
  • for all PO-based translations
    • update PO files (if either the language itself or en has updates)
    • generate XML files (if the language has updates)

If there are is any build failure in the en version, the daily build will be aborted. This is mainly to avoid such breakage to leak into the PO files and thus affect any PO-based translations. The same goes when there are errors generating PO or XML files (can happen if an invalid PO file is committed by a translator). See the next section on how to deal with this and other breakage.

  • 'svn ci' any changes (in POT and PO files) from the previous steps
  • for all other languages that have updates
    • build the language
    • if successful, move to output directory
    • upload its log file to alioth

For translations, build failures in the PDF or text versions will not prevent the HTML version from being uploaded.

  • generate statistics and index page
  • upload anything that was built to alioth
  • remove lockfile
  • mail language log files to translators (unless build was aborted) if log upload was successful only a link will be included, if not the log will be attached
  • mail overview log file to faw :-)

The log directory will contain:
- main.log - overview log file

  • <lang>.log - detailed build log for a specific language
  • svn_up.log - log for svn update; parsed to check for updates
  • svn_ci.log - log for svn commit
  • po-stats.log - log file from po_stats/gen
  • tr-stats - output file from XML-based translation stats

Build failures

Build failures (of output files) in individual languages will not abort the build as a whole and fixing them is in principle the responsibility of translators. However, in some cases (especially failures in PDF builds), they may need some help. In my experience managing the daily builds will give you plenty of experience in debugging build errors :-)

Don't know where to look next?X

If the build is really aborted for any reason, a file do_not_build will be created in the $ROOT directory. This will prevent any other builds from starting until you have deleted it manually (after fixing the cause of the build failure).

After a build failure, you should in principle always restart the build manually with the --retry (or --restart, or -r) option. If a build is restarted, the SVN logs are not deleted. Instead the new 'svn up' or 'svn ci' are appended to whatever was left after the previous build, which ensures that all translations that had updates will be built again.

Basically, the whole build will be done again and any commits to D-I SVN that were done in the mean time (hopefully including the fix for the build failure) will be included.

Managing the builds

This interacts somewhat with release management, so I'll keep it limited for now.

One task - that should probably be shared among editors of the original (English) version, but may fall to you by default - is adding notices about important changes affecting translators in 'index.changes'. I'm not quite sure if translators actually check these, but well... Changes worth being listed include:
- addition of new sections

  • major changes in existing text (especially if text is moved unchanged)
  • changes in the use of optional entities (build/entities/l10n/) or in translatable entities (build/lang-options)
  • ... (It would probably make sense to remove most items that predate the Etch release.)

The other task is (de)activating languages for the daily builds, which is done in bin/languagelist.
Note that for official builds there are separate archlist and langlist files in the debian/ dir of the manual. This is mostly because other criteria are used to activate a language for the development version than for the official version.

There are three possible states for a language:
- full builds - "Y" in column 'Build'

  For languages that are maintained and fairly complete.
- single builds (only i386) - "S" in column 'Build'
  For languages that are very outdated (more that a release behind)   and languages that are very incomplete (new translations).   These languages are listed in a separate section on the index page.
- disabled - line commented out

  For completely abandoned/aborted translations. These languages are not   listed on the index page at all.

Confused? Frustrated?X

Otherwise, just be careful when making changes in the build system. In my experience seemingly innocent changes often have unexpected consequences. That should not prevent you from making improvements if you see them though.

Hope this is enough to get you started.

In practice all this is not all that much work. Builds aborting happens only rarely (but always when preparing a release...). Most work is in release management, not in the daily builds. Still, if you feel this is too much, now is your last chance to back out :-)

Cheers,
FJP [1] My ancient PIII server was unreliable for a while and then I got another box (not a server) that could run the builds a lot faster, so I kept them manual.
[2] http://d-i.alioth.debian.org/manual/

-- 
To UNSUBSCRIBE, email to debian-boot-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org



    

  • application/pgp-signature attachment: stored
Received on Sat Jun 30 13:37:12 2007

This archive was generated by hypermail 2.1.8 : Sat Jun 30 2007 - 13:40:03 EDT

Contact Us  Legal Notices  Order Services Online 
Pantek Home  Privacy Policy  IT news  Site Map  Pantek Library