SVN to GIT migration Using svn2git

We already had plans to migrate SVN to GIT for a while now in EM, but we just never found a chance to do it. Last week we had a major outrage in our Primus data center, which hosts all our development tools, including JIRA, SVN, Bamboo, and all the VM machines for development purposes.

The outrage took quite a few days to recover, and as a result,  our head of engineering decided to migrate all our  SVN repository to GIT in BitBucket.

The tool we used is called svn2git, it provides very easy command line tool for us to check out all the files under a repository, and then import all histories into GIT in no time, well, depends on the size of your repository of course.

This tool provides several features that allow you to include either tags or branches, or simply only trunk, and also allow you to exclude some files based on regular expression patterns. It is quite a handy tool for this kind of migration. Sample commands includes:

svn2git http://svn.example.com/path/to/repo
svn2git http://svn.example.com/path/to/repo --trunk dev --tags rel --nobranches
svn2git http://svn.example.com/path/to/repo --trunk trunk --nobranches --notags
svn2git http://svn.example.com/path/to/repo --rootistrunk
svn2git http://svn.example.com/path/to/repo --exclude doc --exclude '.*~$'
svn2git http://svn.example.com/path/to/repo/nested_project --no-minimize-url

For further installation and usage instructions, please refer to their github home page.

Make Use of Subversion Changelist

During my development time, I constantly need to only commit partial of all my changes in the current project directory and leave the rest of files untouched in order to fix some problems. Every time I do a “svn diff” or “svn commit”, I will have to specify the list of files that have changes in order to view diff or commit them. Luckily subversion comes with a handy feature that helps you in this senario.

Subversion’s changelist support allows you to create changelists by applying labels to files you want to be associated with that changelist, remove those labels, and limit the scope of the files on which its subcommands operate to only those bearing a particular label. To use this feature, simply issue the following commands:

$ svn changelist minor-fixes file-1.php file-2.php
Path 'file-1.php' is now a member of changelist 'minor-fixes'.
Path 'file-2.php' is now a member of changelist 'minor-fixes'.

Then “svn status” will tell you the changes:

$ svn status

--- Changelist 'minor-fixes':
M      file-1.php
M      file-2.php
$

To see diff:

$ svn diff --changelist minor-fixes

And when you commit, can you apply the same options:

$ svn ci -m "Fix minor bug and leave the rest un-touched"  \
--changelist minor-fixes

Then you can continue working with the rest of the files in the current project.

Subversion Externals Basic

I have been using subversion since I started in Hitwise more than a year ago, and have been using externals on the day to day basics, but I have never tried to figure out how to setup externals on the subversion server myself. I have recently bought a new virtual server for my website and installed subversion server myself. Now it is time for me to learn how to configure and maintain the subversion server.

My projects are divided into different directories in the repository, it would be easier to manage the whole project by using externals so that one single checkout will get everything we needed. Here’s a quick howto on using subversion externals to load other subversion repositories into various directories of your project.

Firstly you need to setup the SVN_EDITOR environment variable by running the following command:

[pre]
$ export SVN_EDITOR=vim
[/pre]

Vim is my favourite editor, you can choose whatever editor you prefer.

I have created an “externals” directory under the root repository, so that all externals will be stored there for easy management.

Now create a new directory for you project, for my case, I run

[pre]
$ svn mkdir http://example.com/repo/externals/blog -m “create blog project externals”
[/pre]

Now checkout the externals directory

[pre]
$ svn co http://example.com/repo/externals/blog
[/pre]

Change to the new directory “blog” just created by the checkout

[pre]
$ cd blog
[/pre]

Now run the command to create external rules

[pre]
$ svn propedit svn:externals .
[/pre]

Now you can make changes to the externals of the current directory specified by ‘.’, you can specify a path-url combination per line, then save and exit your editor to finish the change. The path part comes first, and is relative to where your setting the external. Example of the format:

[pre]
include http://example.com/repo/trunk/blog/include
public_html http://example.com/repo/trunk/blog/public_html
php_include http://example.com/repo/trunk/php_include
[/pre]

Once your done adding externals, you can run “svn update” to actually pull down the external, you’ll need to commit to actually push the external property change to the server so anybody else working on the project gets it too.

[pre]
$ svn update
$ svn commit -m “Added external for blog project”
[/pre]

There, any changes made to the other repository or path will come down when you do svn update. You can also commit changes within that external’s directory by just entering that and doing normal subversion commands.

Happy subversion!!

Per-Directory Access Control in SVN

Now you have SVN successfully installed on your Ubuntu server, how do you manage the access control of your SVN repository? This article explains this in more detail.

In the article “Install SVN with Web Access on Ubuntu”, I used the this configuration:

[pre]

DAV svn

SVNPath /svn

AuthType Basic
AuthName “Subversion Repository Login Required”
AuthUserFile /etc/apache2/dav_svn.passwd
Require valid-user

Require valid-user

[/pre]

in file /etc/apache2/mods-enabled/dav_svn.conf. This configuration simply gives users the read and write access to the repository, as long as he/she provides the right username and password, it has no control over who should have read, and who should have write access to the repository.

In order to have a better control over access, we need to add a new line in this configuration file, to tell SVN to read the access rules from a certain file. Add the following line to the configuration file:

[pre]
AuthzSVNAccessFile /etc/apache2/dav_svn.access
[/pre]

Save the file and then open the the access file /etc/apache2/dav_svn.access, actually, it can be any file name and any location you like, add the following rules:

[pre]
[/]
* = r

[:]
user1 = rw
user2 = r
user3 =
[/pre]

If file /etc/apache2/dav_svn.access contains nothing, it simply tells the SVN server to deny any access to the repository. However, you probably need to give at least read access to the root repository to all users, the first two lines in the new file does exactly this. “[/]” means the root directory, “*” means all users and “r” means read access.

As I said before, we need to have more control over the repository access, meaning, not every one can write to the repository, and possibly sometimes need to deny read access to certain directories in the repository. The next few lines in the above example show exactly this. Replace the with your repository name, and with the directory path you want to have access control. If we replace it with [svn:/trunk], it simply tells the SVN server that give user1 read and write access to the trunk directory under root repository, give user2 only read access, and deny user3 any access at all.

Very simple, isn’t it? Now go ahead and configure your SVN server yourself. Remember, since it is an Apache/WebDAN configuration, it requires Apache restart before your configuration takes effect.

SVN Repository Transfer

I recently had to move my main Subversion repository to a new server, so I thought I would pass along this quick how-to.

To move a Subversion repository from one system to another you only have to enter a couple of easy subversion commands. To start, go to the source system and at a command prompt or terminal window type:

[pre]
svnadmin dump /path/to/repository > repository-name.dmp
[/pre]

If the dump file is rather large you can compress it with your favorite zip utility. Now you need to get the dump to your new server, so simply transfer the file via sFTP.

Once the dump file is on the new machine and uncompressed, you need to set up and load the new repository by typing:

If the repository is not created yet, type in

[pre]
svnadmin create new-repository-path
[/pre]

then load the dump file

[pre]
svnadmin load new-repository-path < repository-name.dmp
[/pre]

Of course, you will need to be in the direcotry where repository-name.dmp file resides.

A couple of small things to note – the dump file will be rather large as it represents every commit made on your repository. If your repository is rather large and mature, this file could get quite large. So I think this approach works well for non-large svn repository, and it becomes impractical for very large repositories, e.g. the repositories that are more than a few hundred MB big. Also this method works across platforms so moving from UNIX to Windows or visa-versa is also possible.