How to maintain Git

Communication (45%)

Mailing list discussions on general design, fielding user
questions, diagnosing bug reports; reviewing, commenting on,
suggesting alternatives to, and rejecting patches.

Integration (50%)

Applying new patches from the contributors while spotting and
correcting minor mistakes, shuffling the integration and
testing branches, pushing the results out, cutting the
releases, and making announcements.

Own development (5%)

Scratching my own itch and sending proposed patch series out.

Feature releases are numbered as vX.Y.0 and are meant to contain bugfixes and enhancements in any area, including functionality, performance and usability, without regression.

Maintenance releases are numbered as vX.Y.Z (0 < Z) and are meant to contain only bugfixes for the corresponding vX.Y.0 feature release and earlier maintenance releases vX.Y.W (W < Z).

The master branch is used to prepare for the next feature release. In other words, at some point, the tip of master branch is tagged as vX.(Y+1).0, when vX.Y.0 is the latest feature release.

maint branch is used to prepare for the next maintenance release. After the feature release vX.Y.0 is made, the tip of maint branch is set to that release, and bugfixes will accumulate on the branch, and at some point, the tip of the branch is tagged with vX.Y.1, vX.Y.2, and so on.

next branch is used to publish changes (both enhancements and fixes) that (1) have worthwhile goal, (2) are in a fairly good shape suitable for everyday use, (3) but have not yet demonstrated to be regression free. Reviews from contributors on the mailing list help to make the determination. After a topic is merged to next, it is tested for at least 7 calendar days before getting merged to master.

seen branch is used to publish other proposed changes that do not yet pass the criteria set for next (see above).

The tips of master and maint branches will not be rewound to allow people to build their own customization on top of them. Early in a new development cycle, next is rewound to the tip of master once, but otherwise it will not be rewound until the end of the cycle.

Usually master contains all of maint and next contains all of master. seen contains all the topics merged to next, but is rebuilt directly on master.

The tip of master is meant to be more stable than any tagged releases, and the users are encouraged to follow it.

The next branch is where new action takes place, and the users are encouraged to test it so that regressions and bugs are found before new topics are merged to master.

When a problem is found in a topic in next, the topic is marked not to be merged to master. Follow-up patches are discussed on the mailing list and applied to the topic after being reviewed and then the topic is merged (again) to next. After going through the usual testing in next, the entire (fixed) topic is merged to master.

One release cycle for a feature release is expected to last for eight to ten weeks. A few "release candidate" releases are expected to be tagged about a week apart before the final release, and a "preview" release is tagged about a week before the first release candidate gets tagged.

After the preview release is tagged, topics that were well reviewed may be merged to master before spending the usual 7 calendar days in next, with the expectation that any bugs in them can be caught and fixed in the release candidates before the final release.

After the first release candidate is tagged, the contributors are strongly encouraged to focus on finding and fixing new regressions introduced during the cycle, over addressing old bugs and any new features. Topics stop getting merged down from next to master, and new topics stop getting merged to next. Unless they are fixes to new regressions in the cycle, that is.

Soon after a feature release is made, the tip of maint gets fast-forwarded to point at the release. Topics that have been kept in next are merged down to master and a new development cycle starts.

Scan mailing list. Respond with review comments, suggestions etc. Kibitz. Collect potentially usable patches from the mailing list. Patches about a single topic go to one mailbox (I read my mail in Gnus, and type \C-o to save/append messages in files in mbox format).

Write his own patches to address issues raised on the list but nobody has stepped up to solve. Send it out just like other contributors do, and pick them up just like patches from other contributors (see above).

Review the patches in the saved mailboxes. Edit proposed log message for typofixes and clarifications, and add Acks collected from the list. Edit patch to incorporate "Oops, that should have been like this" fixes from the discussion.

Classify the collected patches and handle master and maint updates:

Obviously correct fixes that pertain to the tip of maint are directly applied to maint.

Obviously correct fixes that pertain to the tip of master are directly applied to master.

Other topics are not handled in this step.

This step is done with "git am".

$ git checkout master    ;# or "git checkout maint"
$ git am -sc3 mailbox
$ make test

In practice, almost no patch directly goes to 'master' or
'maint'.

Review the last issue of "What’s cooking" message, review the topics ready for merging (topic→master and topic→maint). Use "Meta/cook -w" script (where Meta/ contains a checkout of the todo branch) to aid this step.

And perform the merge.  Use "Meta/Reintegrate -e" script (see
later) to aid this step.

$ Meta/cook -w last-issue-of-whats-cooking.mbox

$ git checkout master    ;# or "git checkout maint"
$ echo ai/topic | Meta/Reintegrate -e ;# "git merge ai/topic"
$ git log -p ORIG_HEAD.. ;# final review
$ git diff ORIG_HEAD..   ;# final review
$ make test              ;# final review

Handle the remaining patches:

Anything unobvious that is applicable to master (in other words, does not depend on anything that is still in next and not in master) is applied to a new topic branch that is forked from the tip of master (or the last feature release, which is a bit older than master). This includes both enhancements and unobvious fixes to master. A topic branch is named as ai/topic where "ai" is two-letter string named after author’s initial and "topic" is a descriptive name of the topic (in other words, "what’s the series is about").

An unobvious fix meant for maint is applied to a new topic branch that is forked from the tip of maint (or the oldest and still relevant maintenance branch). The topic may be named as ai/maint-topic.

Changes that pertain to an existing topic are applied to the branch, but:

obviously correct ones are applied first;

questionable ones are discarded or applied to near the tip;

Replacement patches to an existing topic are accepted only for commits not in next.

The initial round is done with:

$ git checkout ai/topic ;# or "git checkout -b ai/topic master"
$ git am -sc3 --whitespace=warn mailbox

and replacing an existing topic with subsequent round is done with:

$ git checkout master...ai/topic ;# try to reapply to the same base
$ git am -sc3 --whitespace=warn mailbox

to prepare the new round on a detached HEAD, and then

$ git range-diff @{-1}...
$ git diff @{-1}

to double check what changed since the last round, and finally

$ git checkout -B @{-1}

to conclude (the last step is why a topic already in 'next' is
not replaced but updated incrementally).

Whether it is the initial round or a subsequent round, the topic
may not build even in isolation, or may break the build when
merged to integration branches due to bugs.  There may already
be obvious and trivial improvements suggested on the list.  The
maintainer often adds an extra commit, with "SQUASH???" in its
title, to fix things up, before publishing the integration
branches to make it usable by other developers for testing.
These changes are what the maintainer is not 100% committed to
(trivial typofixes etc. are often squashed directly into the
patches that need fixing, without being applied as a separate
"SQUASH???" commit), so that they can be removed easily as needed.
The expectation is that the original author will make corrections
in a reroll.

By now, new topic branches are created and existing topic branches are updated. The integration branches next, jch, and seen need to be updated to contain them.

If there are topics that have been merged to master and should be merged to maint, merge them to maint, and update the release notes to the next maintenance release.

Review the latest issue of "What’s cooking" again. Are topics that have been sufficiently long in next ready to be merged to master? Are topics we saw earlier and are in seen now got positive reviews and are ready to be merged to next?

If there are topics that have been cooking in next long enough and should be merged to master, merge them to master, and update the release notes to the next feature release.

If there were patches directly made on maint, merge maint to master; make sure that the result is what you want.

$ git checkout master
$ git merge -m "Sync with 'maint'" --no-log maint
$ git log -p --first-parent ORIG_HEAD..
$ make test

Prepare to update the jch branch, which is used to represent somewhere between master and seen and often is slightly ahead of next, and the seen branch, which is used to hold the rest.

$ Meta/Reintegrate master..jch >Meta/redo-jch.sh

The result is a script that lists topics to be merged in order to
rebuild the current 'jch'.  Do the same for 'seen'.

Review the Meta/redo-jch.sh and Meta/redo-seen.sh scripts. The former should have a line # match next---the idea is that merging the topics listed before the line on top of master should result in a tree identical to that of next.

As newly created topics are usually merged near the tip of seen, add them to the end of the Meta/redo-seen.sh script. Among the topics that were in seen, there may be ones that are not quite ready for next but are getting there. Move them from Meta/redo-seen.sh to the end of Meta/redo-jch.sh. The expectation is that you’d use jch as your daily driver as the first guinea pig, so you should choose carefully.

Now we are ready to start rebuilding jch and merging topics to next. For each branch whose tip is not merged to next, one of three things can happen:

The commits are all next-worthy; merge the topic to next;

The new parts are of mixed quality, but earlier ones are next-worthy; merge the early parts to next;

Nothing is next-worthy; do not do anything.

This step is aided with Meta/redo-jch.sh script created earlier.
If a topic that was already in 'next' gained a patch, the script
would list it as "ai/topic~1".  To include the new patch to the
updated 'next', drop the "~1" part; to keep it excluded, do not
touch the line.

If a topic that was not in 'next' should be merged to 'next', add
it before the '### match next' line.  Then:

$ git checkout --detach master
$ sh Meta/redo-jch.sh -c1

to rebuild the 'jch' branch from scratch.  "-c1" tells the script
to stop merging at the first line that begins with '###'
(i.e. the "### match next" line you added earlier).

At this point, build-test the result.  It may reveal semantic
conflicts (e.g. a topic renamed a variable, another added a new
reference to the variable under its old name), in which case
prepare an appropriate merge-fix first (see appendix), and
rebuild the 'jch' branch from scratch, starting at the tip of
'master', this time without using "-c1" to merge all topics.

Then do the same to 'next'.

$ git checkout next
$ sh Meta/redo-jch.sh -c1 -e

The "-e" option allows the merge message that comes from the
history of the topic and the comments in the "What's cooking" to
be edited.  The resulting tree should match 'jch^{/^### match next'}'
as the same set of topics are merged on 'master'; otherwise there
is a mismerge. Investigate why and do not proceed until the mismerge
is found and rectified.

If 'master' was updated before you started redoing 'next', then

$ git diff 'jch^{/^### match next}' next

would show differences that went into 'master' (which 'jch' has,
but 'next' does not yet---often it is updates to the release
notes).  Merge 'master' back to 'next' if that is the case.

$ git merge -m "Sync with 'master'" --no-log master

When all is well, clean up the redo-jch.sh script with

$ sh Meta/redo-jch.sh -u

This removes topics listed in the script that have already been
merged to 'master'.  This may lose '### match next' marker;
add it again to the appropriate place when it happens.

Rebuild seen on top of jch.

$ git checkout -B seen jch
$ sh Meta/redo-seen.sh

When all is well, clean up the redo-seen.sh script with

$ sh Meta/redo-seen.sh -u

Double check by running

$ git branch --no-merged seen '??/*'

to see there is no unexpected leftover topics.

At this point, build-test the result for semantic conflicts, and
if there are, prepare an appropriate merge-fix first (see
appendix), and rebuild the 'seen' branch from scratch, starting at
the tip of 'jch'.

Update "What’s cooking" message to review the updates to existing topics, newly added topics and graduated topics.

This step is helped with Meta/cook script.

$ Meta/cook

This script inspects the history between master..seen, finds tips
of topic branches, compares what it found with the current
contents in Meta/whats-cooking.txt, and updates that file.
Topics not listed in the file but are found in master..seen are
added to the "New topics" section, topics listed in the file that
are no longer found in master..seen are moved to the "Graduated to
master" section, and topics whose commits changed their states
(e.g. used to be only in 'seen', now merged to 'next') are updated
with change markers "<<" and ">>".

Look for lines enclosed in "<<" and ">>"; they hold contents from
old file that are replaced by this integration round.  After
verifying them, remove the old part.  Review the description for
each topic and update its doneness and plan as needed.  To review
the updated plan, run

$ Meta/cook -w

which will pick up comments given to the topics, such as "Will
merge to 'next'", etc. (see Meta/cook script to learn what kind
of phrases are supported).

Compile, test and install all four (five) integration branches; Meta/Dothem script may aid this step.

Format documentation if the master branch was updated; Meta/dodoc.sh script may aid this step.

Push the integration branches out to public places; Meta/pushall script may aid this step.

Each topic is tested individually, and also together with other topics cooking first in seen, then in jch and then in next. Until it matures, no part of it is merged to master.

A topic already in next can get fixes while still in next. Such a topic will have many merges to next (in other words, "git log --first-parent next" will show many "Merge branch ai/topic to next" for the same topic.

An unobvious fix for maint is cooked in next and then merged to master to make extra sure it is Ok and then merged to maint.

Even when next becomes empty (in other words, all topics prove stable and are merged to master and "git diff master next" shows empty), it has tons of merge commits that will never be in master.

In principle, "git log --first-parent master..next" should show nothing but merges (in practice, there are fixup commits and reverts that are not merges).

Commits near the tip of a topic branch that are not in next are fair game to be discarded, replaced or rewritten. Commits already merged to next will not be.

Being in the next branch is not a guarantee for a topic to be included in the next feature release. Being in the master branch typically is.

Due to the nature of "SQUASH???" fix-ups, if the original author agrees with the suggested changes, it is OK to squash them to appropriate patches in the next round (when the suggested change is small enough, the author should not even bother with "Helped-by"). It is also OK to drop them from the next round when the original author does not agree with the suggestion, but the author is expected to say why somewhere in the discussion.