Getting Started With Git
So, you think you know git? Take this [quiz] to get permission to work with the kent source code git repository.
Gitting Started With Git
(This page is intended for UCSC Genome Browser developers and staff.)
Git is a modern (SCM) source code management system written by Linus Torvalds.
Setting Up Your Own Personal Git Kent Repository
To create your personal git repository of the kent source, please use the following simple directions (note, you can also copy someone's .gitconfig file, changing the name and email, but the below steps will reproduce):
cd $HOME git config --global user.name "Your Name Here" git config --global user.email yourlogin@ucsc.edu
To enable automatic merge without requiring a comment (the way git used to be):
tcshell uses this syntax (see .tcshrc):
setenv GIT_MERGE_AUTOEDIT no
bash uses this syntax:
GIT_MERGE_AUTOEDIT=no export GIT_MERGE_AUTOEDIT
As an alternative, this option does not work globally, so you run it in every git repo that you use:
git config core.mergeoptions --no-edit
If you like colors: (warning this may require particular terminal settings to work right)
git config --global color.diff auto git config --global color.status auto git config --global color.branch auto
[and if you want to turn it off?]
If you are security-conscious:
chmod 644 ~/.gitconfig
Turn off a nagging message about not having a default value set for git push.
git config --global push.default simple
These steps are not needed, but historical:
If you have an existing old test local git repo ~/kentgit, please remove it before proceeding (the first mv step is only applicable for those with cvs versions).
mv kent kent-cvs # move your old kent directory out of the way git clone yourlogin@hgwdev.gi.ucsc.edu:/data/git/kent.git/ kent cd kent
These steps get you the repository desired: /data/git/kent.git is our shared kent repository. We access it via SSH. Here are the summary git clone commands:
- git clone yourlogin@hgwdev.gi.ucsc.edu:/data/git/kent.git/ kent
- git clone yourlogin@hgwdev.gi.ucsc.edu:/data/git/htdocsExtras.git/ htdocsExtras
- git clone yourlogin@hgwdev.gi.ucsc.edu:/data/git/genecats.git/ genecats
- git clone yourlogin@hgwdev.gi.ucsc.edu:/data/git/hgdownload.git/ hgdownload
We also have a post-receieve hook sending your pushes to a redmine clone repo. To make sure that it will work, you need to ssh to the redmine machine one time, and answer "y" to the question.
ssh redmine # answer y to any questions exit # close the shell on redmine, we don't need to do anything with it
Making git pulls without passwords: For new people they need to add hgwdev to their authorized_keys. They create some keys and then add a line to their .ssh directory authorized_keys file. Create the key if it doesn't exist:
- ssh-keygen -t rsa
Add the public key to the authorized_keys
- cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Then you should be able to do a git pull and not be asked for your password.
Some simple real-world git usage
Here is an example of some git commands used to modify a file, check on it, diff it, check it in, push it up to shared repo:
vi doSomethingCool.csh git diff --help git diff # show detailed diff between working dir and staged git diff --stat # show condensed view of the names of files that changed git diff --name-status # show only the names of files that changed
git diff --cached # show detailed diff between staged and HEAD of current branch git diff HEAD # show detailed diff between working dir and HEAD of current branch git diff master origin/master # show detailed diff between local master and central master
git status # another way to see info about changes
git add doSomethingCool.csh # notice that this adds it our list of things
# to be committed together in the next commit.
# Whether it is a new file, or just a change, you do a git add.
# Several related things can and should be committed at the same time as a unit.
git commit -m 'made some useful changes ...' # Commits to your local repo only.
# Make the comments useful.
git diff # check that the changes got committed
git status # another way to see info about changes and so on
git fetch # make sure things are up to date including origin/* refs
git diff origin/master # diffing work-dir to the shared-repo
git push # push my change up to be shared with all
# works great for default branch master or for tracking branches
git push origin HEAD:master # push head of my branch to central repo master branch
# Only do this if your branch is related to the central repo master branch
git diff origin/master # verify that there are no more outstanding changes
git status # check info again
Warning:
git commit -a # DO NOT USE. Commits ALL changes in your local-repo (dangerous!)
-a automatically adds ANY and ALL changes in your local-repo to the "cache/commit index" list of things to commit for existing tracked files. Unlike CVS, it is not influenced by your current directory location. Any tracked files that you have modified, for instance common.mk will get checked in.
instead use:
git add . -u # this gets all tracked changes in current dir and subdirs.
# do not forget the -u or it will add ALL files including untracked files.
Sharing Changes With Others
Git is a distributed SCM so it works a bit differently from CVS. Each user has their own local project repository which includes the full history of all changes ever made. This allows one to work offline and had other advantages besides. However, for simplicity there is a shared repository that people push changes to from their local repository. More complex configurations are possible, such as hierarchical. Probably the shared repository approach is fine for our group.
git pull
equivalent to cvs up -dP, this pulls in changes by others. But not if you have removed a file. Those files will not be recovered.
git add somefile; git commit; git push
equivalent to cvs commit, only do this when your changes are ready to share with others.
If you are working on a development branch that does not have tracking set up,
you can use these commands to push and pull to master, but
ONLY IF IT MAKES SENSE TO DO SO.
git pull origin master
# ONLY do this if your branch is a recent relative of the central repo master.
# Note that this does not update "origin/*" including origin/master,
# so a plain git fetch will still be needed if you wish to
# do git log or git diff against origin/master
git push origin HEAD:master
# ONLY do this if your branch is a recent relative of the central repo master.
"origin" refers to the shared repository from which your local repo was cloned.
Stash
Git stash is handy when you are not keeping your sandbox clean with other methods such as using development branches, and you are doing something potentially dangerous such as pulling, merging, or switching branches. Git refuses to lose your stuff due to failures in these operations, so you may be required to use git stash to save away those changes you are not ready to commit yet.
Git stash supports several operations:
git stash # save to stash stack git stash list # list all the stashes you have git stash show <name> # show --stat level details about stash git stash show -p <name> # show diff details about stash git stash save <name> # save a named stash git stash apply <name> # apply a named stash, i.e. merge it in git stash drop <name> # delete the stash git stash pop # apply the stash and clear it from the stack git stash clear # be careful, wipes out all of your stashes
If when pulling or merging you get an error saying "not uptodate", this is usually because you have changes in your working directory that are in danger of being lost because they are not checked in anywhere.
Should recovery from a failed merge get ugly, you might lose your edits and git refuses to be responsible for that.
Newer versions of git have an improved error message that advises you to commit or stash your working dir changes.
If this error occurs, here are your options:
1. Check it in. If it's ready, don't be afraid of commitment.
git add myfile git commit -m 'new great thing' git pull # note that you may now see a regular merge conflict to be resolved.
2. Use stash.
git stash git pull git stash pop # Note that the pop is a merge and you see a conflict to be resolved. # This only restores the working directory. # You can use git stash apply --index to restore staging too.
3. Check it in on another development branch.
git checkout -b brandNewTopic git add myfile git commit -m 'new great thing' git checkout master # now I can pull or merge # the myfile changes are no longer in the working dir. git pull
4. Abandon the changes.
# I really didn't want those changes anyway git checkout myfile # over-write myfile with last committed version on HEAD. # can alternatively use git reset commands with caution, see the section on reset. git pull
5. Copy it aside and restore with checkout or reset(yuck).
# Not recommended. You can't simply do a unix mv, # because git will think you deleted the file and that # this deletion is just another unchecked-in change. cp myfile myfileX # If you try to restore later, BEWARE losing of other peoples changes!!! git checkout myfile # over-write myfile with last committed version on HEAD. # can alternatively use git reset commands with caution, see the section on reset. git pull
When git makes a stash it is really a commit. The git stash command is primarily a convenience feature. The stash works roughly equivalent to:
git checkout -b mystash0 git add -u # add all tracked but dirty changes git commit -m 'my stash 0' git checkout master # return to master branch, or whatever branch you were on. # note that there are no more dirty files on either branch
The real stashes are stored in a separate namespace from tags and branches.
Git stash apply is roughly like:
git merge mystash0 git reset HEAD^
CVS Equivalents
More equivalents here
cvs add somefile ==> git add somefile cvs commit -m 'comment' somefile ==> git add somefile; git commit -m 'comment'; git push cvs log somefile ==> git log somefile You may find it useful to pipe some outputs into the "tig" utility. cvs ann somefile ==> git blame somefile You can see who did what when. cvs up -dP ==> git pull (git fetch only pulls in new objects but does no merging) cvsup ==> git status (but nicely git status does not need to update and thereby mess with your working dir to do it) cvs rm somefile ==> git rm somefile git mv somepath newpath (cvs has no real equivalent)
git diff
git diff does a lot of things. You can see just names or full details. You can diff between different specific commits, between branches, between repositories, between your sandbox and your commit-list, between your commit-list and the head, etc.
Ignoring files
Configuring git to ignore certain files here
Terminology
We say sandbox or working-dir interchangeably here. We also say commit-list, stage, cached, or index, but they all mean the same thing. This is the changes you have said you want in the next commit, but that commit has not been frozen in yet.
Comments
Please use good comments. This is often all people have to go on when looking through the git log commit history. Try to make it a meaninful one-liner if you can. It's worth taking a moment to get it right. And with git, you can fix the comment that is messed up before pushing to central repo and shared history.
git commit --amend
Branches
Read this wiki page on Working with branches in Git for details on how we use branches within our group.
Your default branch in your own repository is called master. Because we imported cvs history, we all have a lot of branches already. There is also a master or head branch on the shared-repository.
You can and should easily create additional branches in your local repository. This requires NO TAGGING, and it's fast and convenient. You can switch back and forth between the master branch for a quick fix and some more involved detailed development branch, or make a quick branch to test some idea, or another friends code. It's cheap to leave these local branches, they don't clog up the shared repository, and you can also clean up ones that you no longer need. Merging stuff between branches is usually pretty easy and smooth. Note that if you have outstanding changes that would be lost when switching branches, you can tuck them away with the git stash command. Then you should be able to switch branches. But you need to later use your stash and delete it to tidy up.
Tree-ish
(Do not be alarmed. You are not experiencing perl hell. Breathe deeply.)
You can use a hashId from a commit. You can use a symbol such as a tag or a branch-tag. You can use tree-ish commands: Remember that merge commits have two parents, the first parent is the mainline branch master^1, the second parent is the other merged-in branch master^2. master^2^1 means the first parent of the second parent of master. master^1 == master^ master^^^ == master~3 master^ == master~ == master^1 == master~1 You can use this to choose ancestors relative to a branch-tag or other symbol or SHA1-id. Use git rev-parse <some-tree-expression> to actually resolve a tree-ish expression into a specific hash-id.
Git Diff and Git Log
git diff and git log (and other commands too) use tree-ish. But they use it in different ways. git diff x..y means give me the diff from x to y (but not including x itself). It simply does a diff between those two commit-endpoints. It does not consider the exact history between them. Notice that git diff y..x is the inverse of x..y so that insertion becomes deletion etc. Unlike git diff, git log is concerned with all the history between the two points. git log x..y means git log ^x y. ^x means not in x (^==NOT==EXCLUSION) Notice that the caret(^) is on the LEFT. So git log ^x y means y and its ancestors not including x and its ancestors. But git log y..x is nothing like git log x..y git log y..x means git log ^y x which means x and its ancestors not including y and its ancestors. So x..y means very different things to diff and log. There is a x...y (triple dots) also for both diff and log. In the case of git log, x...y means all things that are in x and its ancestors and y and its ancestors, but not in any of their common ancestors. git diff x...y means find common ancestor of x and y, and then diff from there to y.
Here is a handy diff if you have been working on a branch, and you do not want to have to do a git pull from master, you can use this method to see only the changes ON YOUR BRANCH, and ignore changes on master since your branch started: git diff --stat origin/master...HEAD git log --stat ^origin/master HEAD
Re-basing
Don't go crazy with re-basing. It's not usually necessary.
Another important rule is, do not change shared history.
Re-basing is a special technique used in your local repo before you push your changes to the shared repo. It allows you make the history appear tidier and more linear. For instance, you may have been working and checked in 3 small closely related changes. You want to just have them all be one single commit with a good comment. Re-basing is one way.
Re-basing can also be used to linearize the history, which is sometimes helpful for making a nice change list. It basically takes your changes since your branch was forked off, updates all your changes and re-writes them as if they had been patched in after the current shared state. This creates a simpler merge and makes reading the shared history easier. There is some fluidity to changes in your own repo, but once it gets pushed up to the shared repo, it's not so easy to change, in part because everyone's history would have to be modified at that point.
Quick Repo Updates
CVS update was getting very slow because the source had grown to thousands of files and CVS update has to check each one for changes. With git, a change log is kept and only new changes need to be processed. This is usually very fast.
No undetected corruption
Git is also big on making sure digital corruption does not creep in, and it will detect it automatically if it happens. Everytime that any object is accessed in git, it runs the SHA1 has on the extracted content and compares it to its ID. If any corruption has occurred, either accidentally or intentionally, you will be informed right away.
No more hanging locks
No more concerns about hanging CVS locks, for example:
cvs log somefile | more
You are working in your own repo, and have full access to all the history. You are not holding up anybody else.
Mirror site access to git repository
Mirror sites will have read-only access here:
git://genome-source.soe.ucsc.edu/kent.git
If you have firewall issues, this will also be provided:
http://genome-source.soe.ucsc.edu/kent.git
Browse the kent source online
http://genome-source.soe.ucsc.edu/gitlist/
Make a permanent URL link
Make a permanent link to the latest version of a file in the kent source
http://genome-source.soe.ucsc.edu/gitlist/kent.git/raw/master/src/makefile
External git Documentation
The official git home page, documentation, and tutorial. Another interesting tutorial.
A recommended codecademy session about git.
A good complete resource for git is the book "Pro Git" by Scott Chacon. The complete text of the book is available online here.
Here is a link you can access to a book via our VPN library connection. Merge Conflict
Here is a GitHub repository to clone in two directories where you can model a git hub conflict (you won't be able to push unless you have been added as a contributor -ask BrianL- but you can model everything up to that and still see git conflict messages): https://github.com/brianleetest/testGit/blob/master/README.md
Installing git on a Mac OS
See also: Installing git for other operating systems.
$ sudo port install git-core
That seems pretty straight forward, but the problem is you may not have the 'port' command installed yet. That is a little more involved. To get this "MacPorts' system installed, please follow their installation procedures at: Guide MacPorts. This could be an extensive procedure depending upon what you may not yet have installed on your Mac since it needs the X11 system and the Xcode tools installed to get MacPorts installed. But once you have this 'port' command, you can install a vast array of interesting software quite easily.
Like Windows, the default Mac disk is case-insensitive. If there are two versions of the file with different cases, this will cause problems. Google for work-arounds.
Genome Browser Mirror Sites
Move your existing cvs kent directory out of the way:
mv kent kent.cvs
Start a new kent git repository. This clone command will establish a new kent directory:
git clone git://genome-source.soe.ucsc.edu/kent.git
If you have a firewall that interferes with that operation, use http:
git clone http://genome-source.soe.ucsc.edu/kent.git
Then mark your repository with the beta tag so it will track along with our beta releases. Important: you need to be in the newly created kent directory for this command to function:
cd kent git checkout -t -b beta origin/beta
On other versions of git, this may be:
git checkout --track -b beta origin/beta
Some older versions of git do not allow this tracking option.
Resolving merge conflicts in Git
Galt's tips for avoiding merge conflicts
1. Check your stuff in before merge/pull whenever possible. You will thank yourself for it. You can make a branch if you need to.
2. Sometimes git will not allow you to start the pull/merge because it knows you will lose unchecked in stuff. So then you check it in or stash it.
3. ONCE THE MERGE HAS BEGUN AND FAILED, it shows an error, then you need to realize that you are in a kind of critical state where you want to be careful not to lose other people's changes.
- You need to edit each file which git status shows as "merge".
- You need to git add the file. Yes, really, just do it!
- Repeat until merge conflicts are all gone.
- git commit -m 'good message'
!!!! Do not try to specify a filename here.
i.e. DO NOT DO: git commit -m 'msg' somefile(s)
Hopefully git will not allow it, realizing that you are in the middle of a merge.
You are committing EVERYBODY ELSE'S changes along with your merge-conflict resolution. Do not lose their changes. Their changes are sitting in your sandbox and staging area. Get the conflicts resolved, git add them, then do the commit for everything involved in the merge.
You want to commit it ALL.
(It would be helpful to say "Merge" somewhere in your commit message too.)
ESPECIALLY DO NOT DO DURING A MERGE FAILURE:
git stash // DO NOT DO !!!
just messes up your merge. Too bad git does not disallow it.
git reset // DO NOT DO !!!
this screws up your staging area, clearing it and
LOSING other peoples changes, while leaving your sandbox
full of files from the messed-up merge
that will be a pain to find and remove
to recover when you finally fix what's wrong.
If you have goofed up and suspect you might have lost other people's changes, STOP AND GET HELP. DO NOT GIT PUSH.
I spent an hour this morning going over how this happened to someone and showing them what they needed to do right instead, which was quite easy when you know what you're doing.
NOTE that once you have safely and correctly gotten past the merge resolution and checkin, you are no longer in a critical-path, so to speak, and git use can return to normal.
IF you get yourself messed up, I can help you recover.
Yes, there is a way to abandon a merge, but it's not that much fun. It's almost always better to just resolve the conflicts, add then, commit. Go forward! You can do it! Ganbatte!
