Integrated solution for setting up private Debian repository

The motivation

People are using Ubuntu. People are not using Debian. Why?

I believe one reason is that Canonical has set up Launchpad PPA for Ubuntu. It is a good platform for third-party developers to build personal Ubuntu repository and provide their own softwares. When we look back to the upstream, Debian, it seems that Debian is not providing any third party solutions. Yes theoretically you may push any software into Debian official repository through sponsorship and mentor process, but it is hard due to high requirements of Debian project. Why not set up a private Debian repo and have everything in control?

TODO

Setting up mini-buildd

Introduction

Mini-buildd is an integrated tool that can handle various jobs related to debian packaging together on one system. To be more concrete, this tool includes:

• a central daemon which can coordinate various jobs
• a web server based on Django as web frontend
• an ftp daemon which can accept incoming upload (ftpd, act as ftp-master)
• use sbuild to set up chroot environment and finish building works
• use reprepro to generate and maintain Debian repository

This tool is developed inside Debian project, and the git repo is kept on alioth.

Installation

# apt install mini-buildd

During the installation, debconf will ask for the password for admin user. Keep this password securely.

Note that this package is experiencing FTBFS on stretch and sid due to incompatibility with Django 1.10. You must install it on Jessie with debian-backports.

Configuration

Please follow upstream documentation (http://mini-buildd.installiert.net/extra/mini-buildd/documentation/user.html). The documentation is also shipped with mini-buildd. You may find shipped document in /usr/share/doc/mini-buildd or from web frontend.

Using web frontend

Initially the Django daemon will listen on localhost:8066. To make it publicly available or secure (e.g., use HTTPS), you may set up some reverse proxy using nginx or apache2.

NOTE: The web frontend is currently poorly designed and will only accept standalone domain name. For example, setting up a reverse proxy with domain name buildd.example.com is acceptable, but using mydomain.com/buildd/ will surely experience problems.

Initialisation

Follow quickstart guide in upstream documentation.

Stop before trying to build the test package or keyring, since gnupg is not installed on stretch or sid chroot. This looks like a bug.

To solve this problem, chroot into those environment and install gnupg manually with apt. The files are placed under /var/lib/mini-buildd. Bug solved in version 1.0.13+.

P.S. I am also wondering if we need to update base chroot environment manually. Restarting mini-buildd.service will trigger automatic chroot upgrade.

If all set, try to build keyring package and test packages. If keyring building is ok and test packages are built as expected, you may go ahead and try to set up uploading environment.

Managing mini-buildd

TODO

Uploading package as a package maintainer

Uploading is very much alike official Debian package uploading. That said, you need to use dput to upload to a ftp daemon. The ftp daemon is set up using python, and will listen to port 8067 by default.

Write proper packaging scripts

TODO

Read Debian New Maintainers' Guide. Different language versions are available.

Preparing source package or binary package

OK. First of all you must understand WHAT YOU ARE DOING WHEN DOING AN UPLOAD.

You must upload a source package of the software. This may includes a .dsc file, a .orig.tar.gz file and a .debian.tar.gz file. Those files, together with .changes file, help mini-buildd to build a proper .deb file of certain architecture.

You may upload some binary package, aka .deb file. Several years ago, Debian developers have to upload binary package for every architecture together with source package, since the building process is not automated on the server. After the establishment of buildd network, a distributed building automation system is set up, and all the building process can be finished automatically. From then on, it is not necessary for the developers to upload binary packages for every architecture, since packages for the missing binary architectures will be built on server side. However, it is recommended to build at least one binary package (a common example is amd64 package) locally because you may prevent FTBFS(Failed To Build From Source) problem in this way.

Preparing source package

Various tools are available. The most traditional way is to use dpkg-source. Yet we highly recommend the uploader to use a high-level tool and integrate the generation process within building process using debuild, pbuilder, cowbuilder, git-buildpackage or gbp-buildpackage. One-stop solution is always better, isn't it?

If you really want to make a SourceOnlyUpload, please take a look at the article on Debian Wiki.

Using gbp-buildpackage

When you want to upload both a source package (*.dsc) AND one binary package (e.g., the amd64 binary package), the easist way is to use gbp. Note that the building process of binary package will take place LOCALLY, so make sure you have installed some proper package on local workstation, such as git-pbuilder, cowbuilder, devscripts, etc.

This method is the most fashionable way to build a git-managed software.

I will not discuss about the usage of git-buildpackage(aka gbp). Only showing a most commonly building command here, run it in the top directory of git repository:

gbp buildpackage --git-pbuilder --git-upstream-branch=upstream -sa

Change the parameters according to your setup. -sa option will make sure that the original tarball is always regenerated. Of course, you have to install proper backend for building

Doing upload with dput

Settings on mini-buildd server

To simplify the process, you may enable anonymous uploading on mini-buildd. This will raise a warning, but not very critical, since uploaded package must be signed with the uploader's gpg key. If not or the signature is invalid, the uploaded files will be rejected, so no unauthorised uploading will take place. However, all uploaded files will be stored. This may cause some problem if the disk is limited. Maybe we need to delete them manually?

Before real setup, we also have to register gpg public key of the uploader on the server. You may find it in the Config page of mini-buildd web frontend.

TODO

Settings on client side

Please install python-mini-buildd and dput on your client* side, i.e., your local workstation. Generate you dput configuration using python-mini-buildd is simple, just follow the guide in documentation.

NOTE: every time you changed the server side configuration (e.g., enabling upload verification), you MUST regenerate dput configuration to fit the new uploading requirements.

Doing real upload

If everything is set up correctly, the uploading is simple. Follow the guide in dput man page.

NOTE: Your upload may be REJECTED even if everything is uploaded onto ftpd. Various reasons exists, so make sure to check Daemon Logs on web frontend as admin and read the log related to REJECT reasons.

As for mini-buildd, your package will be easily rejected if the following things are not set:

• The distribution written in debian/control must be the same as uploading repo. This cannot be simply unstable, but something like sid-<reponame>-unstable.
• The version in debian/control must be something like (.*)~testSID+1. Actually I cannot understand what the developers of mini-buildd is thinking.
• The uploading must contain an original tarball.