Processing 40 TB of code from ~10 million projects with a dedicated server and Go for $100
2019/09/30 (13129 words)

The command line tool I created Sloc Cloc and Code (scc) (which is now modified and maintained by many other excellent people) counts lines of code, comments and makes a complexity estimate for files inside a directory. The latter is something you need a good sample size to make good use of. The way it works is that it counts branch statements in code. However what does that actually mean? For example “This file has a complexity of 10” is not very useful without some context. To solve this issue I thought I would try to run scc at all the source code I could get my hands on. This would also allow me to see if there are any edge cases I didn’t consider in the tool itself. A brute force Q/A trial by fire.

However if I am going to run it over all that code, which is going to be computationally expensive I may as well try to get some use out of it. As such I decided to record everything as I went and see if I could get something interesting in the end, hence this post.

In short I downloaded and processed a lot of code using scc. The raw numbers include,

• 9,985,051 total repositories
• 9,100,083 repositories with at least 1 identified file
• 884,968 empty repositories (those with no files)
• 3,529,516,251 files in all repositories
• 40,736,530,379,778 bytes processed (40 TB)
• 1,086,723,618,560 lines identified
• 816,822,273,469 code lines identified
• 124,382,152,510 blank lines identified
• 145,519,192,581 comment lines identified
• 71,884,867,919 complexity count according to scc rules
• 2 new bugs raised in scc

Lets get the elephant out of the room first. It was not 10 million projects as the “click bait” title indicates. I was shy by 15,000 so I rounded up. Please forgive me.

It took about 5 weeks to download and run scc over the collection of repositories saving all of the data. It took just over 49 hours to crunch the 1 TB of JSON and produce the results below.

Also note its likely I messed up some of the calculations. I will keep this up to date with the data set provided as this occurs.

Quicklinks

• Methodology
• Presenting and Computing Results
• Cost
• Data Sources
• How many files in a repository?
• Whats the project breakdown per language?
• How many files in a repository per language?
• How many lines of code are in a typical file per language?
• Average complexity for file in each language?
• Average comments for file in each language?
• What are the most common filenames?
• How many repositories appear to be missing a license?
• Which languages have the most comments?
• How many projects use multiple .gitignore files?
• Which language developers have the biggest potty mouth?
• Longest files by lines per language
• Whats the most complex file in each language?
• Whats the most complex file weighted against lines?
• Whats the most commented file in each language?
• How many “pure” projects
• Projects with TypeScript but not JavaScript
• Anyone using CoffeeScript and TypeScript?
• What’s the typical path length, broken up by language
• YAML or YML?
• Upper, lower or mixed case?
• Java Factories
• Ignore files
• Future ideas
• So why bother?
• Raw / Processed Files

Methodology

Since I run searchcode.com I already have a collection of over 7,000,000 projects across git, mercurial, subversion and such. So why not try processing them? Working with git is usually the easiest solution so I ignored mercurial and subversion this time and exported the full list of git projects. Turns out I actually had 12 million git repositories being tracked, and I should probably update the page to reflect that.

So now I have 12 million or so git repositories which I need to download and process.

When you run scc you can choose to have it output the results in JSON optionally saving this file to disk like so scc --format json --output myfile.json main.go the results of which look like the following (done for a single file),

[
  {
    "Blank": 115,
    "Bytes": 0,
    "Code": 423,
    "Comment": 30,
    "Complexity": 40,
    "Count": 1,
    "Files": [
      {
        "Binary": false,
        "Blank": 115,
        "Bytes": 20396,
        "Callback": null,
        "Code": 423,
        "Comment": 30,
        "Complexity": 40,
        "Content": null,
        "Extension": "go",
        "Filename": "main.go",
        "Hash": null,
        "Language": "Go",
        "Lines": 568,
        "Location": "main.go",
        "PossibleLanguages": [
          "Go"
        ],
        "WeightedComplexity": 0
      }
    ],
    "Lines": 568,
    "Name": "Go",
    "WeightedComplexity": 0
  }
]

As a larger example here are the results as JSON for the redis project, redis.json. All of the results below come from this output without any supporting additional data.

One thing to keep in mind is that scc generally categories languages based on extension (except where extension is shared such as Verilog and Coq). As such if someone puts a HTML file with a java extension it will be counted as a java file. Usually this isn’t a problem, because why would you ever do that? But of course at scale it is. It is something I discovered later where some files were masquerading as another.

A while back I wrote code to create github badges using scc https://boyter.org/posts/sloc-cloc-code-badges/ and since part of that included caching the results, I modified it slightly to cache the results as JSON in AWS S3.

With the badge code working in AWS using lambda, I took the exported list of projects, wrote about 15 lines of python to clean the format so it matched my lambda and make a request to it. I threw in some python multiprocessing to fork 32 processes to call the endpoint reasonably quickly.

This worked brilliantly. However the problem with the above was firstly the cost, and secondly lambda behind API-Gateway/ALB has a 30 second timeout, so it couldn’t process large repositories fast enough. I knew going in that this was not going to be the most cost effective solution but assuming it came close to $100 I would have been willing to live with it. After processing 1 million repositories I checked and the cost was about $60 and since I didn’t want a $700 AWS bill I decided to rethink my solution. Keep in mind that was mostly storage and CPU, or what was needed to collect this information. Assuming I processed or exported the data it was going to increase the cost considerably.

Since I was already in AWS the hip solution would be to dump the url’s as messages into SQS and pull from it using EC2 instances or fargate for processing. Then scale out like crazy. However despite working in AWS in my day job I have always believed in taco bell programming. Besides it was only 12 million repositories so I opted to implement a simpler (cheaper) solution.

Running this computation locally was out due to the abysmal state of the internet in Australia. However I do run searchcode.com fairly lean using dedicated servers from Hetzner. These boxes are quite powerful, i7 Quad Core 32 GB RAM machines often with 2 TB of disk space (usually unused). As such they usually have a lot of spare compute based on how I use them. The front-end varnish box for instance is doing the square root of zero most of the time. So why not run the processing there?

I didn’t quite taco bell program the solution using bash and gnu tools. What I did was write a simple Go program to spin up 32 go-routines which read from a channel, spawned git and scc subprocesses before writing the JSON output into S3. I actually wrote a Python solution at first, but having to install the pip dependencies on my clean varnish box seemed like a bad idea and it keep breaking in odd ways which I didn’t feel like debugging.

Running this on the box produced the following sort of metrics in htop, and the multiple git/scc processes running (scc is not visible in this screen capture) suggested that everything was working as expected, which I confirmed by looking at the results in S3.

Presenting and Computing Results

Having recently read https://mattwarren.org/2017/10/12/Analysing-C-code-on-GitHub-with-BigQuery/ and https://psuter.net/2019/07/07/z-index I thought I would steal the format of those posts with regards to how I wanted to present the information. My twist on the previous however is to add jQuery DataTables over the large tables of information. This allows you to sort and search/filter results. As such you can click the headers to sort and use the search box to filter. The search box indicates that this is enabled for you. I also added a jump link near these tables so you can skip over them if you like.

The size of the data I needed to process raised another question. How does one process 10 million JSON files taking up just over 1 TB of disk space in an S3 bucket?

The first thought I had was AWS Athena. But since it’s going to cost something like $2.50 USD per query for that dataset I quickly looked for an alternative. That said if you kept the data there and processed it infrequently this might still work out to be the cheapest solution.

I posted the question on the company slack because why should I solve issues alone.

One idea raised was to dump the data into a large SQL database. However this means processing the data into the database, then running queries over it multiple times. Plus the structure of the data meant having a few tables which means foreign keys and indexes to ensure some level of performance. This feels wasteful because we could just process the data as we read it from disk in a single pass. I was also worried about building a database this large. With just data it would be over 1 TB in size before adding indexes.

Seeing as I produced the JSON using spare compute, I thought why not process the results the same way? Of course there is one issue with this. Pulling 1 TB of data out of S3 is going to cost a lot. In the event the program crashes that is going to be annoying. To reduce costs I wanted to pull all the files down locally and save them for further processing. Handy tip, you really do not want to store lots of little files on disk in a single directory. It sucks for runtime performance and file-systems don’t like it.

My answer to this was another simple go program to pull the files down from S3 then store them in a tar file. I could then process that file over and over. The process itself is done though very ugly go program to process the tar file so I could re-run my questions without having to trawl S3 over and over. I didn’t bother with go-routines for this code for two reasons. The first was that I didn’t want to max out my server, so this limits it to a single core for the hard CPU work (another to read the tar file was mostly blocked on the processor). The second being I didn’t want to ensure it was thread-safe.

With that done, what I needed was a collection of questions to answer. I used the slack brains trust again and crowd-sourced my work colleagues while I came up with some ideas of my own. The result of this mind meld is included below.

You can find all the code I used to process the JSON including that which pulled it down locally and the ugly python script I used to mangle it into something useful for this post https://github.com/boyter/scc-data Please don’t comment on it, I know the code is ugly and it is something I wrote as a throwaway as I am unlikely to ever look at it again.

If you do want to review code I have written to be read by others have a look at the source of scc.

Cost

I spent about $60 in compute while trialling lambda. I have not looked at the S3 storage cost yet but it should be close to $25 based on the size of the data. However this is not including the transfer costs which I also have not observed. Please note I cleared the bucket when I was finished with it so this is not an ongoing cost for me.

However after time I chose not to use AWS in the end because of cost. So what’s the real cost assuming I wanted to do it again?

Well to start all the software used is free as in freedom and open source. So nothing to worry about there.

In my case the cost would be free as I used “free compute” left over from searchcode. Not everyone has compute lying around however. So lets assume another person wishes to replicate this and as such needs to get a server.

It could be done for €73 using the cheapest new dedicated server from Hetzner https://www.hetzner.com/dedicated-rootserver However that cost includes a new server setup fee. If you are willing to wait and poke around on their auction house https://www.hetzner.com/sb you can find much cheaper servers with no setup fee at all. At time of writing I found the below machine which would be perfect for this project and is €25.21 a month with no setup fee.

Best part though? You can get the VAT removed if you are outside the EU. So give yourself an additional 10% discount on top if you are in this situation.

So were someone to do this from scratch using the same method I eventually went with it would cost under $100 USD to redo the same calculations, and more likely under $50 if you are a little patient or lucky. This also assumes you use the server for less than 2 months which is enough time to download and process. This also includes enough time for you to get a list of 10 million repositories to consider processing as well.

If I were to use a gzipped tar file in my analysis (which isn’t that hard to do really) I could even do 10x the repositories on the same machine as the resulting file would still be small enough to fit on the same hard disk. That would take longer to download though which is going to increase the cost for each additional month and this might take multiple months to do.

Going much larger then 100 million repositories however is going to require some level of sharding. Still it is safe to say that you could redo the entire process I did or larger one on the same hardware without much effort or code changes.

Data Sources

From the three sources, github, bitbucket and gitlab how many projects came from each? Note that this is counted before excluding empty repositories hence the sum is over the number of repositories that actually form the counts below this point.

Sorry to the GitHub/Bitbucket/GitLab teams if you read this. If this caused any issues for you (I doubt it) I will shout you a refreshing beverage of your choice should we ever meet.

How many files in a repository?

On to the real questions. Lets start with a simple one. How many files are in an average repository? Do most projects have a few files in them, or many? By looping over the repositories and counting the number of files we can then drop them in buckets of 1, 2, 10, 12 or however many files it has and plot it out.

The X-axis in this case being buckets of the count of files, and Y-axis being the count of projects with that many files. This is limited to projects with less than 1000 files because the plot looks like empty with a thin smear on the left side if you include all the outliers.

As it turns out most repositories have less than 200 files in them.

However what about plotting this by percentile, or more specifically by 95th percentile so its actually worth looking at? Turns out the vast majority 95% of projects have less than 1,000 files in them. While 90% of them have less than 300 files and 85% have less than 200.

If you want to plot this yourself and do a better job than I here is a link to the raw data filesPerProject.json.

Whats the project breakdown per language?

This means for each project scanned if a Java file is identified increment the Java count by one and for the second file do nothing. This gives a quick view of what languages are most commonly used. Unsurprisingly the most common languages include markdown, .gitignore and plain text.

Markdown the most commonly used language in any project is included in just over 6 million projects which is about 2/3 of the entire project set. This makes sense since almost all projects include a README.md which is displayed in HTML for repository pages.

The full list is included below.

skip table to next section

How many files in a repository per language?

An extension of the above, but averaged over however many files are in each language per repository. So for projects that contain java, how many java files exist in that project, and on average for all projects how many files is that?

You can use this to see if a project is larger or smaller than usual for your language of choice.

skip table to next section

How many lines of code are in a typical file per language?

I suppose you could also look at this as what languages on average have the largest files? Using the average/mean for this pushes the results out to stupidly high numbers. This is because projects such as sqlite.c which is included in many projects is joined from many files into one, but nobody ever works on that single large file (I hope!).

So I calculated this using the median value. Even so there are still some definitions with stupidly high numbers such as Bosque and JavaScript.

So I figured why not have both? I did one small change based on the suggestion of Darrell (Kablamo’s resident and most excellent data scientist) and modified the average value to ignore files over 5000 lines to remove the outliers.

skip table to next section

Average complexity for file in each language?

What’s the average complexity per file for each language?

The complexity estimate isn’t really directly comparable between languages. Pulling from the README of scc itself

The complexity estimate is really just a number that is only comparable to files in the same language. It should not be used to compare languages directly without weighting them. The reason for this is that its calculated by looking for branch and loop statements in the code and incrementing a counter for that file.

So comparing languages to each other is not the idea here, although it may be comparable between similar languages such as Java and C for example. Your mileage may vary.

This is more useful when you think about it applying to single files in the same language. So you could answer the question “On average is the file I am working with more or less complex than average?”.

I should mention that I am always looking to improve this calculation and looking for submissions to scc to assist with this goal. Usually it is a case of just adding some keywords to the languages.json file so any programmer of any skill level should be able to assist with this.

skip table to next section

Average comments for file in each language?

What’s the average amount of comments used per file in each language?

You could probably rephrase this to asking what developers write the most comments assuming you squint enough.

skip table to next section

What are the most common filenames?

What filenames are most common across all code-bases ignoring extension and case?

Had you asked me before I started this I would have said, README, main, index, license. Thankfully the results reflect my thoughts pretty well. Although there are a lot of interesting ones in there. I have no idea why so many projects contain a file called 15 or s15.

The makefile being the most common surprised me a little, but then I remembered it is used in many new JavaScript projects. Another interesting thing to note is that it appears jQuery is still king and reports of its death are greatly exaggerated, with it appearing as #4 on the list.

Note that due to memory constraints I made this process slightly lossy. Every 100 projects checked I would check the map and if an identified filename had < 10 counts it was dropped from the list. It could come back for the next run and if there was > 10 at this point it would remain. It shouldn’t happen that often but it is possible the counts may be out by some amount if some common name appeared sparsely in the first batch of repositories before becoming common. In short they are not absolute numbers but should be close enough.

I could have used a trie structure to “compress” the space and gotten absolute numbers for this, but I didn’t feel like writing one and just abused the map slightly to save enough memory and achieve my goal. I am however curious enough to try this out at a later date to see how a trie would perform.

How many repositories appear to be missing a license?

This is an interesting one. Which repositories have an explicit license file somewhere? Note that the lack of a license file here does not mean that the project has none, as it might exist within the README or be indicated through SPDX comment tags in-line. it just means that scc could not find an explicit license file using its own criteria which at time of writing means a file ignoring case named “license”, “licence”, “copying”, “copying3”, “unlicense”, “unlicence”, “license-mit”, “licence-mit” or “copyright”.

Sadly it appears that the vast majority of repositories are missing a license. I would argue that all software should have a license for a variety of reasons but here is someone else’s take on that.

How many projects use multiple .gitignore files?

Some may not know this but it is possible to have multiple .gitignore files in a git project. Given that fact how many projects use multiple .gitignore files? While we are looking how many have none?

What I did find that was interesting was one project that has 25,794 .gitignore files in its repository. The next highest was 2,547. I have no idea what is going on there. I had a brief look at it and it looks like they are used to allow checking in of the directories but I cannot confirm this.

Bringing this back to something sensible here is a plot of the data up to 20 .gitignore files and close to 99% of the total result.

Something you would expect would be that the majority of projects would have either 0 or 1 .gitignore files. This is confirmed by the results with a massive drop-off of 10x for projects with 2 .gitignores. What was surprising to me was how many projects have more than a single .gitignore file. The long tail is especially long in this case.

I was also curious as to why some projects had thousands of .gitignore files. One of the main offenders appears to be forks of https://github.com/PhantomX/slackbuilds which all have ~2,547 .gitignore files. However the other repositories with 1000+ ignore files are listed below.

• 25,794 files https://github.com/knot-git/rrs_publication_ocr
• 1,523 files https://github.com/brunosimon/hetic
• 1,267 files https://github.com/par4all/par4all
• 1,186 files https://github.com/ramonluis1987/code_school
• 1,133 files https://github.com/signed/license-maven-plugin

skip table to next section

Which language developers have the biggest potty mouth?

Working this out is not an exact science. It falls into the NLP class of problems really. Picking up cursing/swearing or offensive terms using filenames from a defined list is never going to be effective. If you do a simple string contains test you pick up all sorts or normal files such as assemble.sh and such. So to produce the following I pulled a list of curse words, then checked if any files in each project start with one of those values followed by a period. This would mean a file named gangbang.java would be picked up while assemble.sh would not. However this is going to miss all sorts of cases such as pu55syg4rgle.java and other such crude names.

The list I used contained some leet speak such as b00bs and b1tch to try and catch some of the most interesting cases. The full list is here.

While not accurate at all as mentioned it is incredibly fun to see what this produces. So lets start with a list of which languages have the most curse words. However we should probably weight this against how much code exists as well. So here are the top ones.

Interesting! My first thought was “those naughty C developers!” but as it turns out while they have a high count they write so much code it probably isn’t that big a deal. However pretty clearly Dart developers have an axe to grind! If you know someone coding in Dart you may want to go offer them a hug.

I also want to know what are the most commonly used curse words. Lets see how dirty a mind we have collectively. A few of the top ones I could see being legitimate names (if you squint), but the majority would certainly produce few comments in a PR and a raised eyebrow.

Note that some of the more offensive words in the list did have matching filenames which I find rather shocking considering what they were. Thankfully they were not very common and didn’t make my list above which was limited to those which had counts over 100. I am hoping that those files only exist for testing allow/deny lists and such.

Longest files by lines per language

As you would probably expect Plain Text, SQL, XML, JSON and CSV take the top positions of this one, seeing as they usually contain meta-data, database dumps and the like.

NB Some of the links below MAY not translate 100% due to throwing away some information when I created the files. Most should work, but a few you may need to mangle the URL to resolve.

skip table to next section

Whats the most complex file in each language?

Once again these values are not directly comparable to each other, but it is interesting to see what is considered the most complex in each language.

Some of these files are absolute monsters. For example consider the most complex C++ file I found COLLADASaxFWLColladaParserAutoGen15PrivateValidation.cpp which is 28.3 MB of compiler hell (and thankfully appears to be generated).

NB Some of the links below MAY not translate 100% due to throwing away some information when I created the files. Most should work, but a few you may need to mangle the URL to resolve.

skip table to next section

Whats the most complex file weighted against lines?

This sounds good in practice, but in reality… anything minified or with no newlines skews the results making this one effectively pointless. As such I have not included this calculation. I have however created an issue inside scc to support detection of minified code so it can be removed from the calculation results https://github.com/boyter/scc/issues/91

It’s probably possible to infer this using just the data at hand, but id like to make it a more robust check that anyone using scc can benefit from.

Whats the most commented file in each language?

Whats the most commented file in each language? I have no idea what sort of information you can get out of this that might be useful but it is interesting to have a look.

NB Some of the links below MAY not translate 100% due to throwing away some information when I created the files. Most should work, but a few you may need to mangle the URL to resolve.

skip table to next section