Celestial Coding

Showcase => Projects => Topic started by: Nathan on July 30, 2013, 09:50:41 pm

Title: srchub - free source code hosting
Post by: Nathan on July 30, 2013, 09:50:41 pm
Sourcehub (https://srchub.org) is a free source code hosting service ran by yours truly.

I currently offer hosting for:

To get a repo just register, login, click "Request project". Then I will need to go in and approve it :).

Sourcehub offers downloads (or also known as releases), issue tracker, wiki, and even code reviews. You can even add other people as contributors. The reason why I am doing this is simply because other free services have a habit of removing features (or shutting down in the case of google code) that you are using from under you - this will never happen at sourcehub.

Sourcehub is a fork of indefero (http://www.indefero.net/) with many bug fixes and new features. And as an olive branch - not only have I released the source code for srchub (something which is not technically required by the GPL), but I have also created a debian installer. So I have nothing to hide and I guarantee no PRISM backdoors to the NSA :). Also the installer will allow you to install either indefero or srchub. The srchub installer is naturally also open source.

12/23/2015 updates

Old thread

To show some evidence that I am somewhat capable of running this service: you can view stats here (http://stats.datanethost.net/). Backups are performed using Backuppc (http://i.imgur.com/kfbPn1x.png) to a readynas with 8TB of disk (http://i.imgur.com/rxOFbxc.png).

Where do I want to go with this?

* Offer jenkins hosting for every project as a continuous integration service
* Integrate OTP to passwords
* Openauth intergration
* Ability to export and import a project

Here are some FAQs:

1. Are you currently encrypting or are you going to encrypt repos?

I'm thinking about this - and currently I am not encrypting repos. This is something I would like to look into - however right now the drives in the KVM hosts are software RAID-1 7200RPM drives. So while I am already taking a hit from the software RAID, I wouldn't want to add another layer of potential bottlenecks. However, if I shoved some SSDs into the servers I think I could very well set that up.

2. Software RAID sucks!!! You should feel bad!

Yes, software RAID is much more resource intensive than hardware RAID but Linux software RAID works so well. For example - if a hardware RAID controller fails - I may have to purchase (or already have an extra) the same controller. With software Linux RAID I can take both drives, put them into another system - reconfigure some stuff like networking and I'm back up and running. I have personally done this before with great success. However SSDs would negate the performance hits (in my opinion).

3. You use backuppc to backup the repos as well?

Currently - yes. Since the repos are not under a high load - the probability of having a half-backup is very remote. I know about the KDE incident - which you can read about here (http://jefferai.org/2013/03/29/distillation/) and here (http://joeyh.name/blog/entry/difficulties_in_backing_up_live_git_repositories/).

4. I don't trust your service because [insert some vague reason here]. Would you assist me in setting up my own sourcehub/indefero instance?

To an extent. I'm not going to SSH to your box and set it up for you, but I will certainly point out documentation and answer questions about any issues you run into.
Title: Re: srchub - free source code hosting
Post by: Nathan on December 21, 2014, 05:06:19 pm
I suppose it's time to bump this topic with updates:

srchub has come a long way since I first "adopted" it. Some of my promises have been fulfilled such as TOTP for login, svn and mercurial authentication (I think I'm the first person to do such a thing), and project exporting (it exports to JSON format - no import yet).

3. is not true anymore - I'm using urbackup now and it's working great. I think backuppc is a great product but it's issue that rather than storing backup metadata in a database - it relies on a listing of files to show available backups. If you have a lot of backups (>1000 folders in same folder) for a specific host this doesn't scale well on a default configured file system. If the disks are local - there are tweaks you can do to the file system to help with this but this appears to be a known issue on many file systems. Unfortunately, my backup mount was a NFS mount so I didn't have the ability to do much. urbackup leverages a database for metadata so it's very fast.

srchub currently has over 100 projects hosted - and hopefully to adopt many more.

I have switched to git for my SCM and finally added it to github - https://github.com/nadams810/srchub

For #4 - please keep in mind that google code appears to be abandoned (https://code.google.com/p/support/issues/list?can=2&q=&sort=-id&colspec=ID%20Type%20Status%20Priority%20Stars%20Owner%20Summary) judging from the spam issues that are getting submitted. The writing was already on the wall when they dropped download releases. So if you don't trust my service but trust google - keep in mind that at any time Google may announce that they will shut down or limit google code even more. As long as I am a developer - srchub isn't going anywhere.

One feature that I would like is to be able to host large projects (like Linux distros or Android ROMs) - but I would need to integrate Amazon S3 support first. I don't have the bandwidth to serve 700MB+ files.
Title: Re: srchub - free source code hosting
Post by: Nathan on October 23, 2015, 11:27:24 pm

I usually don't bring up old threads but I do want to post some updates.

The service has been moved to gandi's infrastructure so now I can support largish files but srchub also supports the ability to link files offsite

Title: Re: srchub - free source code hosting
Post by: Celestialkey on November 27, 2015, 08:51:42 am

I usually don't bring up old threads but I do want to post some updates.

  • OTP has been integrated
  • Exporting a project has been implemented

The service has been moved to gandi's infrastructure so now I can support largish files but srchub also supports the ability to link files offsite

Still no import feature? Sounds like a pretty big flaw if so.
Title: Re: srchub - free source code hosting
Post by: Nathan on December 14, 2015, 11:37:02 pm
Import/conversion is always tricky. Here is where it falls apart - every source data system will be different. Google Code, github, bitbucket, gitlab etc - I could probably spend the next couple of years writing conversion programs to go from that data to srchub - and probably a total of 1.5 people would use it (.5 would be the one person who would use it and decide he didn't want to use srchub).

If I did write a backend process where as if you upload this CSV in this format then I will populate the srchub project - this presents similar problems that the above had. First off - how do I attribute users (for example in FAQs, wiki posts, release uploads etc) that aren't registered users? Call them conversion? Anonymous? This also has the downside of requiring the user to convert their data into this special CSV format. Which is fine for the most part - but it is almost guaranteed that the user will need hand holding when importing their data. Also no one has really inquired about importing of data so I assume that everyone is happy on github.

It's something I want to architect right the first time. And in order to do that I need feedback from people who want to leave their current systems to either the hosted or self-hosted srchub.

As far as export - I wanted the export to allow users to feel like it wasn't a vendor lock in and holding their data hostage. Pretty much the opposite of to Google Code, github and gitlab. As you may know from Google Code - shortly after they announced the shutdown they added the feature to export to github - but they didn't have that before and for years before the close you could tell that Google management didn't care at all about Google Code.

Unfortunately I am the only developer on the project and I'm constantly juggling other projects. Though srchub has come along way from when I first "adopted" it. A little over 2 years and 90 commits. Honestly when I first started to use it my expectation was that within a couple of months I would have to spin up a gitlab instance requiring GBs of RAM for each project. srchub has almost 200 projects and is only actively using 170MB of RAM.