#229 ✓resolved
Chad Woolley

Sometimes Git Hangs

Reported by Chad Woolley | February 27th, 2009 @ 09:23 AM

I just saw this on the ci.rubyonrails.org server. All builds were hung on a 'git fetch origin' process:

ci@ci:~$ ps auxwwg | grep git
ci       28423  0.0  0.0  51012  1720 ?        S    Feb01   0:00 git fetch origin
ci       17610  0.0  0.0  51016  1724 ?        S    Feb02   0:00 git fetch origin
ci       30148  0.0  0.0  51012  1720 ?        S    Feb25   0:00 git fetch origin
ci        4233  0.0  0.0  51016  1720 ?        S    Feb26   0:00 git fetch origin
ci        6956  0.0  0.0  51016  1724 ?        S    Feb26   0:00 git fetch origin

Seems like something should be timing out???

Comments and changes to this ticket

  • Chad Woolley

    Chad Woolley February 27th, 2009 @ 09:30 AM

    Also, when I killed these processes and ran the command manually from the work dir, it worked fine...

    It is also wierd that all of these were running when I had serialize builds set to true.

  • Chad Woolley

    Chad Woolley May 11th, 2009 @ 04:39 PM

    • Assigned user set to “Chad Woolley”
    • State changed from “new” to “resolved”
  • Chad Woolley

    Chad Woolley July 1st, 2009 @ 01:40 PM

    • Assigned user cleared.
    • State changed from “resolved” to “open”
  • Chad Woolley

    Chad Woolley July 1st, 2009 @ 01:43 PM

    Reopening this ticket, git hangs have started happening again:

    sh -c echo /home/ci/.cruise/projects/rails/work ci$ git fetch origin >> /home/ci/.cruise/projects/rails/source_control.err && git fetch origin 2>> /home/ci/.cruise/projects/rails/source_control.err

    This is hard to reproduce, I haven't figure out how to make it happen in a controlled setting. Any advice is welcome...

  • Chad Woolley

    Chad Woolley July 1st, 2009 @ 02:31 PM

    AHA. Once case where this apparently happens if the repo which the project is point at is DELETED. That should be reproducible...

  • Chad Woolley

    Chad Woolley July 27th, 2009 @ 12:07 AM

    Here is another example. I happened to catch this error on the UI, they don't seem to show up in logs anywhere. THis one didn't hang, it crashed the builder (at least I think this is what crashed it):

    /home/woolley/.cruise/projects/geminstaller_using_rubygems_1-1-1/work woolley$ git fetch origin github.com[0: 65.74.177.129]: errno=Connection refused
    fatal: unable to connect a socket (Connection refused)

  • Drew Olson

    Drew Olson August 4th, 2009 @ 09:38 PM

    I'm seeing the same behavior mentioned on connection refused. I have two instances of cc.rb and on only one of them the builder will crash if it ever receives a refused connection from github (which unfortunately happens a lot). In these cases the only fix I have found is to totally bounce cc.rb, which is a pain in the ass. I'll try to keep digging on this and see if I can provide some failing specs, etc.

  • Chad Woolley

    Chad Woolley August 15th, 2009 @ 06:29 PM

    • Assigned user set to “Chad Woolley”
  • Chad Woolley

    Chad Woolley August 15th, 2009 @ 06:31 PM

    Sometimes it crashes the whole builder with an error like this: "

    /home/ci/.cruise/projects/thewoolleyman-rails/work ci$ git fetch origin
    github.com[0: 65.74.177.129]: errno=Connection refused
    fatal: unable to connect a socket (Connection refused) 
    

    This seems to happen all the time on Rails itself, rarely on other projects. Maybe it is the size of the repo on github?

    I attempted a band-aid here:

    http://github.com/thoughtworks/cruisecontrol.rb/commit/88c38476d383...

  • Chad Woolley

    Chad Woolley August 25th, 2009 @ 12:09 AM

    I tried to reproduce the problem locally by forcing a 'git fetch origin' failure (by hacking a bad hostname into the .git/config remote entry), and found an unrelated bug.

    This commit fixes a problem which could cause any polling failure to crash the builder since the Rails 2.3 upgrade, and rolls back the earlier band-aid (which didn't help):

    http://github.com/thoughtworks/cruisecontrol.rb/commit/a14d4f373db6...

    See related ticket:

    https://cruisecontrolrb.lighthouseapp.com/projects/9150-cruise-cont...

  • Chad Woolley

    Chad Woolley September 20th, 2009 @ 09:11 PM

    • State changed from “open” to “resolved”
  • Chad Woolley

    Chad Woolley September 28th, 2009 @ 10:58 PM

    • State changed from “resolved” to “open”

    This is actually due to timeout's broken behavior on MRI 1.8. See http://ph7spot.com/articles/system_timer

    Many thanks to Matt Duncan for helping track this down...

  • Chad Woolley

    Chad Woolley October 4th, 2009 @ 11:12 PM

    The correct way to fix this problem is with SystemTimer (http://ph7spot.com/articles/system_timer).

    However, it is a native gem (and justifiably, that is the point of SystemTimer).

    So, how should we install/use it, given that CCRB currently freezes everything in SCM? We have to install the gem at runtime, but automated installation is hard to get right.

    I'm thinking of using Bundler (http://github.com/wycats/bundler) and triggering the expand/install during server startup. I'll try to make the install fail-safe so the server will still start even if bundler fails, and make the usage of SystemTimer likewise fall back to timeout.rb if it is not installed.

    Let me know if someone has a better idea.

  • Chad Woolley

    Chad Woolley October 5th, 2009 @ 02:08 AM

    Hmm, bundler wasn't as easy to bootstrap as I'd hoped. Opened a bug: http://github.com/wycats/bundler/issues/#issue/68

  • Johnathon Wright

    Johnathon Wright October 23rd, 2009 @ 06:27 AM

    This seems related but frankly I haven't dug in to the code at all to know for sure. But it is irritating. If unrelated, let me know. I'll dig in to it more and create another ticket.

    CruiseControl.rb 1.4.0
    running on Vista (don't laugh)

    [error] undefined method []' for nil:NilClass Build loop failed<br/> NoMethodError: undefined method[]' for nil:NilClass
    ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/source_control/git.rb:76:in current_branch' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/command_line.rb:86:incall' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/command_line.rb:86:in e' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/command_line.rb:84:inpopen' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/command_line.rb:84:in e' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/command_line.rb:71:inexecute' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/command_line.rb:70:in chdir' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/command_line.rb:70:inexecute' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/source_control/abstract_adapter.rb:42:in execute_in_local_cop y'<br/> ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/source_control/abstract_adapter.rb:41:inchdir' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/source_control/abstract_adapter.rb:41:in execute_in_local_cop y'<br/> ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/source_control/git.rb:109:ingit' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/source_control/git.rb:75:in current_branch' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/source_control/git.rb:68:innew_revisions' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../lib/source_control/git.rb:55:in up_to_date?' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../app/models/change_in_source_control_trigger.rb:11:inbuild_necess ary?'
    ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../app/models/project.rb:234:in build_necessary?' c:/ruby/lib/ruby/1.8/fileutils.rb:1029:inany?' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../app/models/project.rb:234:in each' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../app/models/project.rb:234:inany?' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../app/models/project.rb:234:in build_necessary?' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../app/models/project.rb:211:inbuild_if_necessary' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /../config/../app/models/polling_scheduler.rb:13:in run' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /builder:79 ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /builder:78:incatch' ./lib/cruise_control/../../script/../config/../lib/cruise_control/../../script /builder:78 ./lib/cruise_control/../../script/../config/../lib/cruise_control/init.rb:47:i n load'<br/> ./lib/cruise_control/../../script/../config/../lib/cruise_control/init.rb:47:i nbuilder'
    ./lib/cruise_control/../../script/../config/../lib/cruise_control/init.rb:10:i n send'<br/> ./lib/cruise_control/../../script/../config/../lib/cruise_control/init.rb:10:i nrun'
    ./lib/cruise_control/../../script/../config/../cruise:5

  • Chad Woolley
  • Chad Woolley

    Chad Woolley December 17th, 2009 @ 10:12 PM

    • State changed from “open” to “resolved”
    • Tag set to git

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

People watching this ticket

Tags

Pages