Deploying documentation to GitHub pages with Jenkins / Hudson

GitHub Pages is a neat feature of GitHub that allows you to serve static files by using a special gh-pages branch in a repository. Having got Jenkins set up to build my jdbc-persist project, I thought it would be neat to set up something to take the Javadoc generated during a build and automatically push it to the gh-pages branch when the build finishes.

My project uses Maven for building, and I didn't see any way to have the same job that builds the project handle pushing the documentation on a separate branch, so I set up a second job (which I called jdbc-persist-site) to do that.

The process isn't terribly elegant. The first job uses the Maven site:site goal to generate both Javadoc and a whole Maven project site. The second project first pulls my gh-pages branch. It then runs the following batch script (wouldn't be much different as a .sh script):

This basically just grabs the generated site from the other job manually and overwrites the previous version of the files. I actually had to split the three git commands into 3 separate batch file steps in Jenkins because a single script would stop after just 1 git call.

The final step was to push the new commit to gh-pages so that the updated files would show up. This was a simple matter of selecting the Git Publisher post-build action and telling it to publish to the gh-pages branch on origin. I also set up a trigger to run the jdbc-persist-site job after any successful jdbc-persist build, ensuring that any changes I make are automatically reflected on the site each time.

I find an approach like this to be far better than the approach of storing generated Javadoc files in the same branch as the source code the way a number of projects I like (Guava and Guice for example) do. Directories containing artifacts like Javadoc that can be derived from the source clutter up the project structure, clutter up commit history with changes to derived files and add to the volume of files that someone checking out the project has to download. I think the ability to do this sort of thing easily speaks well for the ease and power of Git's branching model and for GitHub's awesome features!

Git clone error on Jenkins/Hudson on Windows

Recently I set up a new instance of Jenkins (formerly Hudson) running on my Windows 7 desktop computer. I tried to set up a job that would pull from a GitHub repository and do a build but (like every other time I've tried this) was foiled by the job simply hanging at the step where it tries to clone or fetch from GitHub. Cancelling the job leads to the following errors in the console output:

Started by user cgdecker
Checkout:workspace / E:\Colin\.hudson\jobs\temp\workspace - hudson.remoting.LocalChannel@692769e1
Using strategy: Default
Checkout:workspace / E:\Colin\.hudson\jobs\temp\workspace - hudson.remoting.LocalChannel@692769e1
GitAPI created
Cloning the remote Git repository
Cloning repository origin
$ git clone -o origin git@github.com:cgdecker/jdbc-persist.git E:\Colin\.hudson\jobs\temp\workspace
ERROR: Error cloning remote repo 'origin' : Could not clone git@github.com:cgdecker/jdbc-persist.git
ERROR: Cause: Error performing git clone -o origin git@github.com:cgdecker/jdbc-persist.git E:\Colin\.hudson\jobs\temp\workspace
null
Trying next repository
ERROR: Could not clone repository
FATAL: Could not clone
hudson.plugins.git.GitException: Could not clone
 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:809)
 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:740)
 at hudson.FilePath.act(FilePath.java:756)
 at hudson.FilePath.act(FilePath.java:738)
 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:740)
 at hudson.model.AbstractProject.checkout(AbstractProject.java:1171)
 at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:499)
 at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:415)
 at hudson.model.Run.run(Run.java:1362)
 at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:405)
 at hudson.model.ResourceController.execute(ResourceController.java:88)
 at hudson.model.Executor.run(Executor.java:145)

First, I tried the usual suspects:

• Ensuring that Jenkins was running as the correct user.
• Ensuring that the user's .ssh directory and keys were in place and checked out.
• Trying the full path to git.exe in Git's bin directory rather than just using git as the path.

I did a search for the error, but as in the past it turned up no solutions that actually fixed the problem for me. I then tried running a job that just executed a Windows batch command to run git clone on the same repository... which worked fine!

The solution

Double-checking things, I noticed that my PATH contained two directories for my Git installation (msysgit): one for Git/bin and another for Git/cmd, which I hadn't been aware of before. Looking in Git/cmd, I noticed that it contained a file git.cmd which appeared to be some kind of script wrapping git.exe. So, I went into the Jenkins configuration and changed the path to the git executable to git.cmd. And that fixed everything!

Based on what I've seen when searching for this error, it seems like there are quite a few potential causes for it. But this is what worked for me in my situation and I haven't seen this solution elsewhere, so I thought I'd write it down... if nothing else to help me remember if it happens again.

Git and CVS

In my last post I talked a little about my experience starting to learn to use Git. Now I'm going to talk about what I'm doing with it currently in an environment where Git can't be used as the end-to-end version control system (at least not immediately).

The Situation

At work we're using CVS for a legacy project that I'm working on. It's been around a while and the practices with it are pretty set. Branches are used for release versions of the product and that's about it. I tend to have a feature or two I'm working on and I also occasionally do something experimental. The trouble is, getting changes related to separate things mixed together can make it confusing to know what needs to be committed and what doesn't when it comes time to commit something (committing early and often would help with this, but it's not quite how things are done). Especially problematic would be if changes related to different things affected the same file.

To deal with this, I'd sometimes check out a separate copy of the project into another directory and work on something specific there, but this wasn't very efficient. I'm programming in Java using Eclipse, and in addition to the issue of keeping track of multiple complete copies of the project, I'd have to switch workspaces or open up another copy of Eclipse to work with those copies. This wasn't very efficient.

The Solution

When I started looking into Git and experimenting with it, I realized that I could probably use it locally with a totally different VCS acting as the central repository. Git stores all its data in a single folder in the root of the working directory, which means that its files aren't mixed into your entire directory structure the way they are with CVS and its CVS subdirectories in every directory.

Here's the process I used to get this project set up for work with Git:

• Check out a clean copy of the project into a new directory to use.
• Get the environment all set up for normal use in Eclipse.
• Do git init to create the Git repository at the root of the project.
• Create and set the .gitignore file, having it ignore compiled code output folders, among other things. (Note: It should also ignore all CVS folders.)
• git add . everything that isn't ignored, making sure you aren't getting anything you don't want to.
• Do the initial git commit.
• From there, use Git normally... create a development branch, and branches off that for features, experiments, etc.
• When you're ready to commit something to CVS, merge the changes all the way down to the master branch and commit in CVS.

When I switch branches, I just refresh the project in Eclipse and it immediately reflects the branch I'm now on. Since the files that exist may differ between branches, I also like to use tasks in Mylyn. I have tasks that are specific to whatever I'm doing on a specific branch, and I activate them while working on them. I'd done a little of that before, but it's even more useful in this situation. If I'm working on some files and need to switch to another branch that doesn't have them, I just deactivate the task and the editor tabs go away. I switch branches, do whatever I need, and then when I switch back and activate the task. Despite the fact that the files ceased to exist on the files system for a while, when the task is reactivated the files are right back there like before!

This approach has made it far easier and more fun to work on multiple separate things at once, and it's easy to do even though the central repository is the ancient CVS! Thanks for being awesome, Git!

Update:

How you interact with CVS when doing this is important. First, the .gitignore file should be set to ignore all CVS folders! You don't want changes to the files in the CVS folders to have to be committed to your Git repository. You want to ignore the fact that you're also using CVS as much as possible. Given that, you should also only ever update from CVS or commit to CVS from your master branch! Things can get really weird if you update or commit in a branch you've created.

Learning Git

One thing that I've been realizing more and more recently is the importance of keeping up with advancements in developer tools. I'm a strong believer that if you aren't using the best tools available for the job you're doing, you aren't being as effective as you could be, regardless of how much effort you put in to it. Plus, as a lazy programmer I certainly don't want to put more effort in to something when a better tool could save me from that.

Up until recently, I'd only used CVS and Subversion for source control... CVS at work and a little in college, and Subversion at home because I knew it was more recent than CVS. As I started to pay more attention to what I was seeing around the web on blogs and Hacker News, I began to notice a lot of positive references to Git. It seemed a little intimidating to me, at first... largely command line based, with no full-featured shell integration like Tortoise for Windows? I didn't know about that... I like just right-clicking on files for diffs, commits, etc.

Tales of the wonders of branching in Git were probably what brought me around to finally try it out this weekend... well, that and GitHub, which is awesome. Anyway, on sitting down and starting to learn it I discovered that it really wasn't hard at all to pick up. A quick git init, git add . and git commit and I had a small existing project that I hadn't put in version control yet committed to a Git repository. And from there, pushing it to a repo on GitHub was incredibly easy as well. I then spent some time experimenting with branching and merging and all that fun stuff. I'll admit that I certainly still don't know all the intricacies of it, but the basic process was as easy and wonderful and rainbows as advertised.

One of the things I really love about Git is the fact that it uses a local repository in addition to a remote one (and that only if you want/need it to). Branches are such a great way of keeping logically separate units of work... well, separate, and that includes experimental changes. It's nice to be able to have experimental branches without them needing to be stored in the central repository that everyone on a project uses. Also great is the concept of the working directory, and how easily and quickly the files in it change as you change branches. Having one directory you can work from with all the different branches of your project has many advantages. I think I'll get in to that and some details of how I'm starting to make use of Git at work in my next post, which will probably be related to using Git locally with a central CVS repository.