https://boyter.org/ Recent content on Ben E. C. Boyter Hugo -- gohugo.io en-au Wed, 31 Jan 2024 00:00:00 +0000 https://boyter.org/posts/supporting-scc/ Wed, 31 Jan 2024 00:00:00 +0000 https://boyter.org/posts/supporting-scc/ Sometime last year Justin Vincent of the techzing podcast suggests that I should add a license to Quote, I “strongly” suggest that you wrap each product into a paid version - where you sell a “license” for usage with priority support. New year means new energy and so I got around to do this. All the tools will remain free software, so there is no need to worry about a rug pull. https://boyter.org/posts/go-libraries-i-like/ Wed, 13 Dec 2023 00:00:00 +0000 https://boyter.org/posts/go-libraries-i-like/ In no particular order a list of Go libraries/packages I really like and some reasons why. zerolog I know that the changes to slog make it far closer to being what zero log is now, but I still prefer it for my own use. requests Anyone who works with python for a while learns about the requests library for making HTTP calls. This is similar but for Go. It has a few sharp edges when dealing with websites that don’t follow standards, but otherwise is amazing. https://boyter.org/posts/who-needs-bastion-hosts/ Sun, 10 Dec 2023 00:00:00 +0000 https://boyter.org/posts/who-needs-bastion-hosts/ So on occasion, I find myself wanting to run SQL against customer databases. Where this gets annoying is that because we are often in AWS using lambda I have no ability to connect to them. Occasionally im able to bully the friendly devops’ and security people to allow me access to a bastion host, which I can then connect to the SQL database. However this is often a painful process if allowed at all, and so with permission I sometimes add a special route into the application allowing me to run arbitrary SQL as needed. https://boyter.org/posts/gopherconsyd-2023/ Thu, 09 Nov 2023 00:00:00 +0000 https://boyter.org/posts/gopherconsyd-2023/ Welcome. If you are reading this you either follow me on RSS, watch my blog or were directed here at the start of my GopherConSyd 2023 talk about “Bloom Filters: Building a Cutting Edge Go Search Engine to Explore the World’s Source Code”. Whatever the case may be, welcome! For the slides visit https://boyter.org/static/gophercon-syd-presentation-2023/ then just use the space bar to advance, the arrow keys to move around or the ESC key to see everything and then enter to select a slide. https://boyter.org/posts/publishing-a-pico8-game/ Thu, 28 Sep 2023 00:00:00 +0000 https://boyter.org/posts/publishing-a-pico8-game/ TL/DR; Play/Download a copy from itch.io I had always wanted to create games. In fact its one of the reasons I went down the development path. For various reasons I never invested enough time (probably too much time playing them) and went into writing code for businesses. In hindsight that was probably the right move from a financial perspective, since it seems most game developers tend to be fairly underpaid and the drama in the game development space makes high school teen drama look tame. https://boyter.org/posts/dataengbytes-2023/ Sat, 26 Aug 2023 00:00:00 +0000 https://boyter.org/posts/dataengbytes-2023/ I was recently selected to present at DataEngBytes 2023 after submitting an idea for a talk based on my blog post Processing 40 TB of code from ~10 million projects with a dedicated server and Go for $100. I think it went well, other than not having the slides able to present for the first 10 minutes. After 5 mins or so I decided to just plough on since it was less of a visual talk anyway. https://boyter.org/posts/walk-code-respecting-gitignore-go/ Mon, 07 Aug 2023 00:00:00 +0000 https://boyter.org/posts/walk-code-respecting-gitignore-go/ Since I maintain a few projects that deal with source code one of the things I needed badly was a way to parse and understand gits .gitignore and .ignore files in order to get as much accuracy as possible. I had previously tried using code I lifted from The Platinum Searcher albeit with some fixes to avoid crashes. Annoyingly however it never implemented glob’s correctly. I tried searching around for another implementation but none appeared to work as expected. https://boyter.org/posts/codespelunker-details/ Tue, 06 Jun 2023 00:00:00 +0000 https://boyter.org/posts/codespelunker-details/ I released code spelunker a few days ago https://github.com/boyter/cs and literally one person asked for details on how it worked. Fitting in with my habit of putting any content I produce into my blog what follows is a built out version of it. So code spelunker started when I noticed someone using Visual Studio to search files recursively in a directory and my own use of ripgrep with fzf. It’s development has been on and off over the last 4 years, roughly broken down into the following categories. https://boyter.org/posts/code-spelunker-a-code-search-command-line-tool/ Mon, 05 Jun 2023 00:00:00 +0000 https://boyter.org/posts/code-spelunker-a-code-search-command-line-tool/ Code spelunker (cs) or code search is a new command line tool I have been working on and off over the last few years. It allows you to search over code or text files in the current directory either on the console, via a TUI or HTTP server, using some boolean queries or regular expressions. I just recently pushed out a stable first version release and thought I would write a quick post about it. https://boyter.org/posts/all-companies-currently-recruiting/ Sun, 21 May 2023 00:00:00 +0000 https://boyter.org/posts/all-companies-currently-recruiting/ I realized I posted the below to LinkedIn some time ago, and since I am in the habit of pulling anything I put on 3rd party platforms back here, well here it is. I am writing this on behalf of all developers/engineers out there. Please stop with the take home coding challenge questions. Really. Just stop it. They are a lazy and frankly an unprofessional way of sorting the wheat from the chaff. https://boyter.org/posts/how-to-start-go-project-2023/ Tue, 16 May 2023 00:00:00 +0000 https://boyter.org/posts/how-to-start-go-project-2023/ I previously wrote about starting a Go project in 2018. A lot has changed since I wrote that and I had been wanting to write an updated version. What follows should be enough for anyone new to Go to get started and ideally start them being productive. Quicklinks Install / Setup Starting a Project Learning Go Searching Building / Installing Linting / Static Analysis / Security Scanning Profiling Unit Testing Integration Testing Community Multiple Main Entry Points OS Specific Code Docker Useful Tools/Packages Install / Setup The first thing to do is download and install Go. https://boyter.org/posts/profiling-ngram-trigram-tokenization-in-go/ Wed, 12 Apr 2023 00:00:00 +0000 https://boyter.org/posts/profiling-ngram-trigram-tokenization-in-go/ Another progress update on https://searchcode.com where I was trying to reduce the time it takes to index source code. Since the this was a real world example of profiling code trying to reduce its cost I thought I would document how it went. The first step is to add code allowing the collection of a profile. I don’t do this often enough in searchcode to have a want to enable/disable this so I just comment/uncomment this when needed. https://boyter.org/posts/deduplicate-slice-go-sort-or-map/ Tue, 11 Apr 2023 00:00:00 +0000 https://boyter.org/posts/deduplicate-slice-go-sort-or-map/ TL;DR: If you don’t need to preserve order of the elements, sort the slice first and dedupe. I have been working on https://searchcode.com a fair bit recently, having been dealing with memory issues which were causing it to crash with out of memory exceptions. The cause of that was due to backtracking regular expressions in the syntax highlighter, which will be the subject of another post sometime. As a result of the above I have spent a fair amount of time looking at memory profiles. https://boyter.org/posts/implement-go-channels-java/ Mon, 13 Mar 2023 00:00:00 +0000 https://boyter.org/posts/implement-go-channels-java/ I have been doing interviews recently, and whenever someone mentions knowing a lot of Go and Java one of the questions I put to them is how to implement Go channels in Java. Since there isn’t anything online to suggest how to do this that I liked reading (that I could find) I thought I would write one, and do it in native Java without needing to import a library. https://boyter.org/posts/activitypub-announce-post/ Mon, 23 Jan 2023 00:00:00 +0000 https://boyter.org/posts/activitypub-announce-post/ I have been investigating how ActivityPub is meant to work. What follows is how a boost/announce/retweet works in Mastodon and any other system that implements against ActivityPub. You can find the full collection on github https://github.com/boyter/activitypub and contribute your own details. ActivityPub -> Announce -> Post When a user clicks on bonk/boost/retweet the following sequence happens. The users intent is converted to an announce. This announce is sent to every follower they have as well as the owner of the post that the user is boosting. https://boyter.org/posts/mastinator-activitypub-breaking-assumptions/ Wed, 28 Dec 2022 00:00:00 +0000 https://boyter.org/posts/mastinator-activitypub-breaking-assumptions/ I didn’t really expect to have a post this late in the year but here we go. Lets go through some assumptions that were broken for me recently. Not just my own through perhaps some other peoples assumptions too. I have been working with ActivityPub for a bit. I want to run my own server, beyond installing some 3rd party. I want to build my own from scratch. However this rapidly exposed some problems with testing ActivityPub. https://boyter.org/posts/how-i-built-my-own-index-for-searchcode/ Tue, 22 Nov 2022 00:00:00 +0000 https://boyter.org/posts/how-i-built-my-own-index-for-searchcode/ Abstract TL/DR I present what I belive is a unique index for indexing and searching source code. It copies ideas from Bing bitfunnel implementation to create a very fast, memory efficient trigram index over source code. searchcode.com is now using a custom built index written by yours truly It indexes 180-200 million documents and 75 billions lines of code The index works using bloom filters sharded by unique document trigrams It borrows some of the core ideas of bitfunnel used in microsoft’s bing The use of trigrams inside a bloom filter search is as far as I can tell unique It lowered the index search times from many seconds to ~40ms across searchcode End user searches still take around 300 ms to process though It also improved search relevance and reliability Architecture wise things became simpler, as the new index sits on a single machine not four It uses caddy as the reverse proxy and redis as a level 2 cache The index sits entirely in memory on a 16 core 5950x CPU and 128 GB RAM machine It currently processes close to a million searches every day I also gave the site a new look and feel which is much better than what it looked like previously I had an absolute blast learning and building it What Happened? https://boyter.org/posts/leet-speak-guid-script/ Mon, 07 Nov 2022 00:00:00 +0000 https://boyter.org/posts/leet-speak-guid-script/ Some time ago I wrote about creating leet speak guid’s as a way of having fun while working with sitecore. https://boyter.org/2014/05/unique-guid/ I recently realised I wanted to have this around so I could create more interesting guids for one reason or another, but couldn’t find the script I was using back then. So I created it again, and its now on github as a gist for myself and anyone else to use https://gist. https://boyter.org/posts/golang-the-clang-command-requires-command-line-developer-tools/ Wed, 28 Sep 2022 00:00:00 +0000 https://boyter.org/posts/golang-the-clang-command-requires-command-line-developer-tools/ Recently on my M1 Macbook I ran into the following issue when trying to compile some Go code with C dependencies. The “clang” command requires the command line developer tools. Being a Mac this also prompted me with a button I could click to install them. Naturally I clicked it, only to be prompted with the following. I can only assume that whoever wrote the Windows file copy dialog estimator now works at Apple as it gave me the following estimates one after another, https://boyter.org/posts/decompiling-java-excel-and-the-vista-tcpip-stack/ Fri, 09 Sep 2022 00:00:00 +0000 https://boyter.org/posts/decompiling-java-excel-and-the-vista-tcpip-stack/ Working for a large corporate company as a mid level developer is generally rather boring. Churn out some CRUD applications. Write ETL as businesses live on data (usually processed in excel). Meetings that are meant to resolve things but just result in more meetings. So in short, it is usually boring. However occasionally you get some sort of interesting problem, especially if you gain a reputation for actually solving them. https://boyter.org/posts/more-annoying-go-bugs/ Mon, 29 Aug 2022 00:00:00 +0000 https://boyter.org/posts/more-annoying-go-bugs/ Thought I would write done some of the more annoying/interesting software bugs I have had to deal with recently. Both were dealing with an archive system where large amounts of video/audio/images are uploaded into, then searchable. It allows the editing of metadata for each type of content, as well sending to external systems. Problem #1 When a record is being edited a flag “This is being edited” is added to the record and removed on save. https://boyter.org/posts/cost-of-integer-cast-in-go/ Mon, 22 Aug 2022 00:00:00 +0000 https://boyter.org/posts/cost-of-integer-cast-in-go/ Recently have been doing interviews at work for Go developers. The filter for this is a simple review exercise. We present a small chunk of code and ask them to review it over 15 minutes pointing out any issues they see. The idea is to respect their and our time. It works pretty well and we can determine how much experience someone has by their ability to pick up the obvious vs subtle bugs. https://boyter.org/posts/if-you-succeed-you-will-fail/ Thu, 21 Jul 2022 00:00:00 +0000 https://boyter.org/posts/if-you-succeed-you-will-fail/ If you succeed, you will fail. Something said at a client engagement I was at some time ago. I figure enough time has passed that it is worth sharing this story. One of the early engagements was to deliver for a very large company an iOS Application powered by a RESTful API. This was problematic due to what is common for every large organisation, a collection of older legacy backends. https://boyter.org/posts/aws-abusing-search-engine-gets-abused/ Sun, 17 Jul 2022 00:00:00 +0000 https://boyter.org/posts/aws-abusing-search-engine-gets-abused/ Some time ago I wrote about building an Australian search engine by abusing how AWS Lambdas work. You can view it at bonzamate.com.au, and read the post about it here. Recently I noticed that Bonzamate.com.au was in turn being abused by a series of bots repeatedly making queries against it. Now normally I don’t worry about bots. There is little you can do to stamp them out entirely, and its always nice to see someone trying to exploit the things you work on. https://boyter.org/posts/search-index-implementations/ Sun, 26 Jun 2022 00:00:00 +0000 https://boyter.org/posts/search-index-implementations/ A living document where I will be putting information about search index implementations as I learn about them. So there are a few talked about ways to search across content or build an index that I am aware of. Lets discuss each in term with its advantages and disadvantages Brute Force Brute Force, note that this isn’t an index strategy per say, but worth discussing anyway. Assuming you can get the entire corpus you are search into RAM it is possible to brute force search. https://boyter.org/posts/testing-running-api-gw-lambda-locally/ Mon, 25 Apr 2022 00:00:00 +0000 https://boyter.org/posts/testing-running-api-gw-lambda-locally/ I have been working with AWS API Gateway and Lambda using Go a lot recently. One of the more annoying things about API Gateway and Lambda is the inability to run things locally. As a result I tend to write a lot of unit tests to compensate. This works up till you start doing things like SQL queries as mocking away the database is a less than ideal situation if you are working with raw SQL, which I tend to do since I don’t like most ORM’s. https://boyter.org/posts/most-incompetent-person-worked-with/ Thu, 10 Feb 2022 00:00:00 +0000 https://boyter.org/posts/most-incompetent-person-worked-with/ Previously I had covered the worst program I had ever worked on (although that probably needs updating now) and the worst individual. I thought it was time branch out into the most incompetent person I ever worked with. So a bit of back history. This was at lendlease where we were working on a system that massaged all employee data into a single place allowing accurate counts of employees, and getting titles mapped to real ones and such. https://boyter.org/posts/installing-voice2json-ubuntu/ Thu, 03 Feb 2022 00:00:00 +0000 https://boyter.org/posts/installing-voice2json-ubuntu/ I have been playing around with offline non cloud tool replacements for doing things like speech to text and the like. One that I found which looked promising was voice2json which is a command line tool for turning speech intent into text. It can do generic conversion too, but a review of that is not the subject of this post. Following the instructions from the website http://voice2json.org/install.html worked fine up till I tried to run it and got the following issue. https://boyter.org/posts/resurrecting-bogan-ipsum/ Tue, 01 Feb 2022 00:00:00 +0000 https://boyter.org/posts/resurrecting-bogan-ipsum/ I remember hearing about Bogan Ipsum, which was an alternative to Lorem Ipsum for you know bogan’s back in 2011 through LifeHacker. However it since became defunct. I became aware of this fact from Darren at work given the following message on slack? Dude Boganipsum where? Clearly this needed to be rectified! A quick search found this https://www.npmjs.com/package/boganipsum implementation which seemed to work well enough, so I ported it to basic JavaScript (think 2001 style JavaScript) and threw together a quick HTML page. https://boyter.org/posts/development-practices/ Wed, 24 Nov 2021 00:00:00 +0000 https://boyter.org/posts/development-practices/ Development Practices Follow a development practice of developer empowerment. Encourage change and experimentation with zero or low penalty for defects allowing them to be found early and resolved with additional tests or automation. I embrace the fact that bugs and problems will be found, but by making the penalty for them exceedingly low we can ensure that they are identified and resolved quickly with little to no production impact. All processes from CI/CD and development are built around this idea and ensure that rapid changes can be made, allowing for extensible solutions, with as low a cost of development as possible while keeping quality high. https://boyter.org/posts/move-mysql-database-between-servers/ Wed, 20 Oct 2021 00:00:00 +0000 https://boyter.org/posts/move-mysql-database-between-servers/ I’m in the middle of moving searchcode.com’s database to another server while upgrading it. This means I need to copy it somehow. It’s especially difficult in this case because the server its sitting on does not have enough disk space to do my usual mysqldump then copy the file across using SCP. I investigated splitting the database using --where clauses allowing me to get chunks of the database across, but that also means adding things like --skip-create-options and --skip-add-drop-table and I ended up having to redo it a few times, with dumps in between. https://boyter.org/posts/abusing-aws-to-make-a-search-engine/ Tue, 14 Sep 2021 00:00:00 +0000 https://boyter.org/posts/abusing-aws-to-make-a-search-engine/ TL/DR I wrote an Australian search engine. You can view it at bonzamate.com.au. It’s interesting because it runs its own index, only indexes Australian websites, is written by an Australian for Australians and hosted in Australia. It’s interesting technically because it runs almost entirely serverless using AWS Lambda, and uses bit slice signatures or bloom filters for the index similar to Bing. I also found out the most successful code I have ever written is PHP, despite never being a professional PHP developer. https://boyter.org/posts/national-broadband-network-a-real-end/ Mon, 23 Aug 2021 00:00:00 +0000 https://boyter.org/posts/national-broadband-network-a-real-end/ A little over a year ago I wrote about my experience getting the NBN and concluding that it was a massive disappointment. I ended with Anyway thats the end of it. A 7 years wait for internet that I guess might be the envy of people living in Libya. Obviously since this post I have something more to say. NBN changed their policy allowing you to get a TCP (technology upgrade) quote without having to pay. https://boyter.org/posts/upgrading-my-old-desktop/ Mon, 09 Aug 2021 00:00:00 +0000 https://boyter.org/posts/upgrading-my-old-desktop/ I used to be really into gaming and even more so building computers, reading CPU reviews, video card reviews, learning about water cooling, what is the best hard disk, psu and such. That was a rather long time ago now. In fact the last time I put any effort into this was the Athlon XP days where I think I had a 2700+ with a Radeon 9700, which was a pretty sick build at the time. https://boyter.org/posts/interesting-go-memory-issue/ Mon, 19 Jul 2021 00:00:00 +0000 https://boyter.org/posts/interesting-go-memory-issue/ So had an interesting experience with Go the other day. This is part of building a bloom filter index which is going to replace the index in searchcode.com So I was trying to verify how long a search would take when the index was full. When trying it out on a machine to see what the performance would be I built a version to create some very large slices over 64,000,000 items in length to simulate the expected workload. https://boyter.org/posts/useful-bootstrap-for-go-projects/ Mon, 07 Jun 2021 00:00:00 +0000 https://boyter.org/posts/useful-bootstrap-for-go-projects/ Mostly here because I keep forgetting all the things I want to put into a Go project when I start it. Collected into a list of things I actually need to do, tools to install and how to get things working. This page is a work in progress and something I will add to as I remember things or discover something new. Static Analysis https://golangci-lint.run/usage/install/ Install that and run golangci-lint run . https://boyter.org/posts/intern-interview-questions/ Sun, 03 Jan 2021 00:00:00 +0000 https://boyter.org/posts/intern-interview-questions/ Recently my workplace has been accepting some interns as part of contributing back to the industry in general, as well as promoting the brand and hopefully being able to grow and grab excellent candidates in the future when they graduate. One recently wanted some time to sit down with me to ask a few questions. In order to give him the best use of time I asked to have the questions written in advance so I could respond to them before chatting and then build out my answers verbally. https://boyter.org/posts/bloom-filter/ Thu, 10 Dec 2020 00:00:00 +0000 https://boyter.org/posts/bloom-filter/ A bloom filter is one of those data structures you are probably already aware of, or have at least heard about. For those looking for a simple recap, they are a probabilistic data structure which can be used to determine if something is in a set or not, giving a slight chance of returning a false positive result for some checks but while using less space than a full hashmap. https://boyter.org/posts/first-pc/ Thu, 12 Nov 2020 00:00:00 +0000 https://boyter.org/posts/first-pc/ At my current workplace we have been doing themed Friday stand-ups. The most recent was about your first PC love, or whatever technology you look back on fondly. What follows is my one. My first PC was I think a Commodore PC40-III. I’m a little fuzzy on the details and its hard to find it based on pictures. Was not my first taste of technology, but the one I look on back most fondly. https://boyter.org/posts/faster-literal-string-matching-in-go/ Wed, 30 Sep 2020 00:00:00 +0000 https://boyter.org/posts/faster-literal-string-matching-in-go/ TL/DR: I wrote a fast literal string matching library in Go get it here https://github.com/boyter/go-string/ Recently I rebuilt searchcode in Go, as mentioned. While there was a variety of reasons for this one was that performance in the previous Python version was not great. Please note this was mostly due to my own shortcomings and not the language itself. However I have always used searchcode as a test bed for interesting problems, since it gets enough traffic to verify things at scale, and I wanted to get better at Go. https://boyter.org/posts/stay-safe-on-the-road/ Sun, 02 Aug 2020 00:00:00 +0000 https://boyter.org/posts/stay-safe-on-the-road/ SI live in an area with a popular driving road. As such you see a lot of people flying around in fancy cars, motorbikes and bike riders. Between where I am and the start there is an awful bit of road that funnels everyone together. As such a lot of people jump the line to get past the bike riders because there isn’t enough space for them to have their own lane (I keep complaining to my local member about this). https://boyter.org/posts/national-broadband-network-a-massive-dissapointment/ Tue, 14 Jul 2020 00:00:00 +0000 https://boyter.org/posts/national-broadband-network-a-massive-dissapointment/ I write this mostly so I can move on. The NBN has been a source of constant frustration and anger for me. While I will never forgive those who caused the current shambles I can perhaps redirect my anger elsewhere will keeping it in the back of my mind. So here is my brief time-line of NBN. I moved to area that was due to get NBN FTTP in ~2014. The fact that it was due to get such a connection influenced the decision. https://boyter.org/posts/unicode-support-what-does-that-actually-mean/ Sun, 14 Jun 2020 00:00:00 +0000 https://boyter.org/posts/unicode-support-what-does-that-actually-mean/ Unicode support in my experience seems to be one of those hand wavy things where most people respond to the question of “Do you support unicode?” with Yeah we support emojis, so yes we support unicode! 😃😭😈 Which is partially correct… It’s certainly a good start. However unicode is more than emojis. So lets take a step back. What is unicode? It’s actually easy to explain, here it is taken from the wiki page “A standard for representing the worlds text”. https://boyter.org/posts/building-an-api-rate-limiter-in-go-for-searchcode/ Mon, 04 May 2020 00:00:00 +0000 https://boyter.org/posts/building-an-api-rate-limiter-in-go-for-searchcode/ So regular readers of this site might know that I run https://searchcode.com which I recently converted over to Go. While I had always run searchcode cheaply I never wanted to implement rate limits and instead operate as Bill and Ted by being “Excellent to each other”. Sadly the internet is not from the Bill and Ted universe and due to some level of abuse against the API I have been forced to implement a rate limiter. https://boyter.org/posts/searchcode-rebuilt-with-go/ Wed, 22 Apr 2020 00:00:00 +0000 https://boyter.org/posts/searchcode-rebuilt-with-go/ So as of about 12 hours a new version of https://searchcode.com/ has been rolled out. It marks the biggest update to the site in over 4 years since I moved over to version 3 which you can read about https://boyter.org/2016/07/searchcode-com-architecture-migration-3-0/ The reason for it was that searchcode has always been something I kept running as a showpiece. It’s something that I can point to publicly and say “I did that”. The idea being to ensure I remain employable. https://boyter.org/posts/the-three-f-s-of-open-source/ Mon, 04 Nov 2019 00:00:00 +0000 https://boyter.org/posts/the-three-f-s-of-open-source/ I write and publish open source code. I quite often get bug fixes, patches, requests and questions regarding the things I have published. Most of the feedback is respectful and helpful. However occasionally there are other requests. This page is where I plan to redirect them as a final warning before I block and ignore them. The three F’s to open source development. Fix it, Fork it, Fuck off. Fix it. https://boyter.org/posts/sloc-cloc-code-can-a-crusty-go-program-outperform-a-well-written-rust-project/ Thu, 31 Oct 2019 00:00:00 +0000 https://boyter.org/posts/sloc-cloc-code-can-a-crusty-go-program-outperform-a-well-written-rust-project/ For those that wish to follow along with the presentation use the below link, https://boyter.org/static/gophercon-syd-presentation/ You can press S to see the speaker notes as you go along. https://boyter.org/posts/elastic-search-mappings-how-to-add-facets-and-aggregations/ Sat, 12 Oct 2019 00:00:00 +0000 https://boyter.org/posts/elastic-search-mappings-how-to-add-facets-and-aggregations/ Developer? Need to know how to use Elasticsearch? Want to know how to get working with it quickly? This is the book for you! Buy now using Leanpub In elasticsearch mappings define type and values in documents. You use them to specify that fields within your document should be treated as numbers, dates, geo-locations and whatever other types elasticsearch supports. You can also define the stemming algorithm used and other useful index fields. https://boyter.org/posts/how-elastic-search-stores-documents/ Thu, 10 Oct 2019 00:00:00 +0000 https://boyter.org/posts/how-elastic-search-stores-documents/ Developer? Need to know how to use Elasticsearch? Want to know how to get working with it quickly? This is the book for you! Buy now using Leanpub It is useful to understand is how elasticsearch stores documents. Documents that are indexed need to go into an index and have a type. Indexes can contain one or more types. This may sound limiting but you can search over all indexes or all types within an index or just one type within an index if you require. https://boyter.org/posts/my-list-of-useful-command-line-tools/ Wed, 09 Oct 2019 00:00:00 +0000 https://boyter.org/posts/my-list-of-useful-command-line-tools/ A post that is constantly a work in progress. ripgrep - Similar to grep, but a drop in replacement for large file scanning without unicode penalities. It’s also similar to Ag and Ack and optimised for searching code, ignoring .git directories and the like. https://github.com/burntsushi/ripgrep tmux - Terminal multiplexer. I wrote a 5 min guide to this. Use it to have resumable sessions after closing ssh sessions and to have split pane windows. https://boyter.org/posts/an-informal-survey-of-10-million-github-bitbucket-gitlab-projects/ Mon, 30 Sep 2019 00:00:00 +0000 https://boyter.org/posts/an-informal-survey-of-10-million-github-bitbucket-gitlab-projects/ The command line tool I created Sloc Cloc and Code (scc) (which is now modified and maintained by many other excellent people) counts lines of code, comments and makes a complexity estimate for files inside a directory. The latter is something you need a good sample size to make good use of. The way it works is that it counts branch statements in code. However what does that actually mean? https://boyter.org/posts/how-aws-and-go-transformed-a-public-broadcaster/ Wed, 03 Jul 2019 00:00:00 +0000 https://boyter.org/posts/how-aws-and-go-transformed-a-public-broadcaster/ Today I presented at the Sydney Dev Warriors meet-up about a project I was working on from the start to end of 2018. In a nutshell it was a self serve digital archive for video, audio, images and metadata. Since I am in the habit of collecting everything I do into one place I chose to do the presentation using reveal.js which means I can also include it here. How AWS and Go Transformed a Public Broadcasters Digital Archive. https://boyter.org/posts/sloc-cloc-code-badges/ Thu, 27 Jun 2019 00:00:00 +0000 https://boyter.org/posts/sloc-cloc-code-badges/ This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links. Sloc Cloc and Code - What happened on the way to faster Cloc 2018-04-16 Sloc Cloc and Code Revisited - A focus on accuracy 2018-08-28 Sloc Cloc and Code Revisited - Optimizing an already fast Go application 2018-09-19 Sloc Cloc and Code a Performance Update 2019-01-09 Sloc Cloc and Code Badges for Github/Bitbucket/Gitlab A very brief update in the world of scc. https://boyter.org/posts/really-cheap-python-chatbot/ Wed, 19 Jun 2019 00:00:00 +0000 https://boyter.org/posts/really-cheap-python-chatbot/ Let me start by stating that I think all chat-bots are rubbish and that a better solution is a decent search implementation. Type in some terms and get some useful results. Much better than typing to a machine. A while ago I was working at a client site where there was an internal discussion group about building a chat-bot. Many meetings had passed, discussing using neural networks, building flows, should we use IBM Watson. https://boyter.org/posts/how-good-does-the-monaro-look/ Tue, 11 Jun 2019 00:00:00 +0000 https://boyter.org/posts/how-good-does-the-monaro-look/ About 12 years ago my old man was working as one of those hire a husband/jim’s maintenance deals. This was in the lower Blue Mountains. During my university break I didn’t have much on and would often come with him as something to do. We were attempting to repair a deck of some flea hole house near Glenbrook. The place would have been improved only by a match and petrol. The owner was around and mentioned that the place was to be rented out and real estate had just sent some people on the way to inspect. https://boyter.org/posts/a-non-gamers-attempt-at-sea-of-thieves/ Mon, 10 Jun 2019 00:00:00 +0000 https://boyter.org/posts/a-non-gamers-attempt-at-sea-of-thieves/ A long weekend just recently passed in NSW Australia. With a larger than normal amount of time on my hands I decided to try out the game Sea of Thieves. Ignoring the fact that the NBN rollout here has ruined my ADSL2+ which has gone from a 18-19 Mbps connection to 3 Mbps and that it took 24 hours to download, what follows is probably one of the more memorable gaming experiences I have ever had. https://boyter.org/posts/file-read-challange/ Wed, 08 May 2019 00:00:00 +0000 https://boyter.org/posts/file-read-challange/ It started with this chain of blog posts, Using Java to Read Really, Really Large Files -> Processing Large Files in Java -> Processing Large Files – Java, Go and ‘hitting the wall’ I thought after reading them I would add to the chain. While I don’t like solving arbitrary company made “programming tests” when doing interviews https://boyter.org/2016/09/companies-recruiting/ whenever I see a post comparing Go to Java where Java is trounced in performance gets my attention. https://boyter.org/posts/developer-empowerment/ Tue, 16 Apr 2019 00:00:00 +0000 https://boyter.org/posts/developer-empowerment/ Recently my job/career path has had me looking “under the hood” at clients development practices. The idea being to see where their strengths and weaknesses are and then make actionable suggestions. Generally every client wants the same outcome. They want to know how to improve software quality and deliver features faster. When doing this I focus on the development side, but honestly a lot of this applies to DevOp’s as well. https://boyter.org/posts/media-clipping-using-ffmpeg-with-cache-eviction-2-random-for-disk-caching-at-scale/ Thu, 04 Apr 2019 00:00:00 +0000 https://boyter.org/posts/media-clipping-using-ffmpeg-with-cache-eviction-2-random-for-disk-caching-at-scale/ A short time ago one of the more interesting blog posts (to me anyway) about cache eviction popped up on Hacker News which prompted me to post the following comment. Love reading this. It has always been one of those interesting things I kept in the back of my mind in my day to day. I was very excited when I actually got to implement it on a real world project. https://boyter.org/posts/my-personal-complaints-about-golang/ Thu, 14 Mar 2019 00:00:00 +0000 https://boyter.org/posts/my-personal-complaints-about-golang/ Go as a language is fairly decent. However because questions about why I have issues with tends to come up often enough on the company slack programming channel (see what I did there?) I figured I would write them down and put it here so I can point people at a link when they ask what my complaints are. For the record I have been using Go heavily for the last year or so, writing command line applications, scc, lc and API’s. https://boyter.org/posts/enough-tmux-to-be-productive-in-5-minutes/ Sun, 24 Feb 2019 00:00:00 +0000 https://boyter.org/posts/enough-tmux-to-be-productive-in-5-minutes/ tmux the terminal multiplexer is one of those brilliant tools I use all day. I am not a tmux expert, nor do I have any need to be. What follows is the bare minimum you need to know to feel productive in tmux, written for some work colleagues. The following assumes you have not configured or tweaked tmux in any way opting for the default settings. From the command line to start or connect to tmux https://boyter.org/posts/why-i-gave-up-on-my-silicon-valley-dream-offer/ Thu, 21 Feb 2019 00:00:00 +0000 https://boyter.org/posts/why-i-gave-up-on-my-silicon-valley-dream-offer/ 2018 was not a good year for me. To clarify that, it was actually the worst year of my life so far. It started well. I had the opportunity and need to learn the programming language Go which I did and then released scc https://github.com/boyter/scc/ written in it as an attempt to prove I have some ability with it. Sales of searchcode server https://searchcodeserver.com/ were picking up, https://searchcode.com/ was getting more attention and some of my blog posts were getting cycled around resulting in more traffic. https://boyter.org/posts/use-golang-to-restore-glacier-to-s3/ Tue, 22 Jan 2019 00:00:00 +0000 https://boyter.org/posts/use-golang-to-restore-glacier-to-s3/ I had a great deal of difficulty searching for an answer to this I have included it here to hopefully save someone else the effort. How to make a S3 request to retrieve items that have transitioned into AWS Glacier from an S3 bucket using the Go AWS SDK API. package main import ( "fmt" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/s3" ) func main() { sess, err := session.NewSession(&aws.Config{ Region: aws.String("ap-southeast-2"), }) svc := s3. https://boyter.org/posts/sloc-cloc-code-performance-update/ Wed, 09 Jan 2019 00:00:00 +0000 https://boyter.org/posts/sloc-cloc-code-performance-update/ Update 2019-03-13 This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links. Sloc Cloc and Code - What happened on the way to faster Cloc 2018-04-16 Sloc Cloc and Code Revisited - A focus on accuracy 2018-08-28 Sloc Cloc and Code Revisited - Optimizing an already fast Go application 2018-09-19 Sloc Cloc and Code a Performance Update 2019-01-09 Sloc Cloc and Code Badges for Github/Bitbucket/Gitlab I thought I had finished with my code counter Sloc Cloc and Code (AKA scc) https://github. https://boyter.org/posts/trimming-golang-binary-fat/ Sun, 06 Jan 2019 00:00:00 +0000 https://boyter.org/posts/trimming-golang-binary-fat/ I have recently started work again on searchcode server with the continued plan to upgrade searchcode to use the same code-base. One big issue with this which I recently tweeted about was that Java has a decent well documented fast library for just about everything, with the exception of a good syntax highlighter like Pygments http://pygments.org/ or Chroma https://github.com/alecthomas/chroma It is however possible to use Pygments in Java using Jython http://pygments. https://boyter.org/posts/expert-excuses-for-not-writing-unit-tests/ Tue, 27 Nov 2018 00:00:00 +0000 https://boyter.org/posts/expert-excuses-for-not-writing-unit-tests/ A while back I collected all of the fake Orly book covers I could find an posted them online https://boyter.org/2016/04/collection-orly-book-covers/ I also mentioned that I was seriously considering writing content for the “Expert Excuses for Not Writing Unit Tests” one. I think I had this idea at the time that it was something I could potentially sell or use to raise my profile. Since I am not a comedy writer there is a market of about 0 people in the universe who would pay for content like this. https://boyter.org/posts/types-of-development/ Sun, 25 Nov 2018 00:00:00 +0000 https://boyter.org/posts/types-of-development/ Types of Development #1 Rage driven development Where an existing application/process causes one to rage so hard you spend the next 6 weeks over-engineering a solution to solve a simple problem. Example Perl. Types of Development #2 Test driven development Where previous projects and death marches result in a violent reaction of over-correcting for past failures and everything must be tested dammit. Example includes any project where the goal is 100% test coverage. https://boyter.org/posts/generating-harry-potter-spells-and-kelewan-houses/ Sun, 28 Oct 2018 00:00:00 +0000 https://boyter.org/posts/generating-harry-potter-spells-and-kelewan-houses/ Occasionally I try to branch out from working on https://searchcode.com/ https://searchcodeserver.com/ or https://github.com/boyter/scc and find something that I find interesting to toy around with. Usually I go for something that will only take a few hours to play around with. I had recently re-read a book series I remember loving a while ago which is the Empire Trilogy of books by Feist and Wurts https://en.wikipedia.org/wiki/Empire_Trilogy One of the things that stuck out to me as I was reading was that the names seemed to be a blend of Korean/Japanese/Chinese/Aztec and seems to follow a standard structure. https://boyter.org/posts/sloc-cloc-code-performance/ Wed, 19 Sep 2018 00:00:00 +0000 https://boyter.org/posts/sloc-cloc-code-performance/ Update 2019-03-13 This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links. Sloc Cloc and Code - What happened on the way to faster Cloc 2018-04-16 Sloc Cloc and Code Revisited - A focus on accuracy 2018-08-28 Sloc Cloc and Code Revisited - Optimizing an already fast Go application 2018-09-19 Sloc Cloc and Code a Performance Update 2019-01-09 Sloc Cloc and Code Badges for Github/Bitbucket/Gitlab I don’t want to make any false claims about the impact of scc and the blog post about it https://boyter. https://boyter.org/posts/sloc-cloc-code-revisited/ Tue, 28 Aug 2018 00:00:00 +0000 https://boyter.org/posts/sloc-cloc-code-revisited/ Update 2019-03-13 This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links. Sloc Cloc and Code - What happened on the way to faster Cloc 2018-04-16 Sloc Cloc and Code Revisited - A focus on accuracy 2018-08-28 Sloc Cloc and Code Revisited - Optimizing an already fast Go application 2018-09-19 Sloc Cloc and Code a Performance Update 2019-01-09 Sloc Cloc and Code Badges for Github/Bitbucket/Gitlab Two things prompted me to start looking at my code counter scc again. https://boyter.org/posts/reading-files-quickly-in-rust/ Mon, 20 Aug 2018 00:00:00 +0000 https://boyter.org/posts/reading-files-quickly-in-rust/ I have been wanting to learn Rust for a while now. I did play with it some time back https://boyter.org/2017/09/working-rust/ for solving some of the project Euler problems and was reasonably impressed with how it turned out. However as I had no practical use for it at the time I ended up investing more time in Go. Go as it turns out is a pretty decent language, and somewhat akin to Python in terms of getting things done. https://boyter.org/posts/why-count-lines-of-code/ Mon, 11 Jun 2018 00:00:00 +0000 https://boyter.org/posts/why-count-lines-of-code/ A work colleague (let’s call him Owen as that’s his name) asked me the other day “I dont understand the problem space scc et al solve. If you wanted to write a short post, i’d read and share the hell out of it. Basically, it seems like a heap of people can see the need for it, and I’m trying to understand it myself” Owen is one of the more switched on people I know. https://boyter.org/posts/how-to-add-multiple-tags-to-go-struct/ Thu, 24 May 2018 00:00:00 +0000 https://boyter.org/posts/how-to-add-multiple-tags-to-go-struct/ I was looking to add multiple tags (the declaration after the struct field) to a struct in Go such that I could have it defined in GORM and in JSON. This was much harder to find via any search engine I tried than I would have guessed. The answer thankfully is simple once you know and was provided to me by Josh of https://boyter.org/2017/03/golang-solution-faster-equivalent-java-solution/ fame. The trick is to define multiple tags but with a space separating them. https://boyter.org/posts/how-to-start-go-project-2018/ Thu, 10 May 2018 00:00:00 +0000 https://boyter.org/posts/how-to-start-go-project-2018/ This post is fairly old. See the updated version Starting a Go project in 2023 Getting started with a Go project in 2018 is frankly a little more painful then getting anything else started IMHO. With that here is what I have been doing to get started. The first thing to do is download and install Go. I would suggest always installing from the Go website itself https://golang.org/ and following the instructions for your OS of choice. https://boyter.org/posts/python-simplehttpserver/ Mon, 23 Apr 2018 00:00:00 +0000 https://boyter.org/posts/python-simplehttpserver/ Something I always forget and have to search for is how to use Python to serve a directory for local testing and the like. Including it below so I have a copy I can always easily look it up. python -m SimpleHTTPServer 8090 and in Python 3 python3 -m http.server https://boyter.org/posts/sloc-cloc-code/ Mon, 16 Apr 2018 00:00:00 +0000 https://boyter.org/posts/sloc-cloc-code/ It started by wanting to write a code counter that would be faster than cloc. Update 2019-03-13 This is now part of a series of blog posts about scc Sloc Cloc and Code which has now been optimised to be the fastest code counter for almost every workload. Read more about it at the following links. Sloc Cloc and Code - What happened on the way to faster Cloc 2018-04-16 Sloc Cloc and Code Revisited - A focus on accuracy 2018-08-28 Sloc Cloc and Code Revisited - Optimizing an already fast Go application 2018-09-19 Sloc Cloc and Code a Performance Update 2019-01-09 Sloc Cloc and Code Badges for Github/Bitbucket/Gitlab TL/DR https://boyter.org/2018/03/quick-comparison-go-file-walk-implementations/ Sun, 11 Mar 2018 09:58:51 +0000 https://boyter.org/2018/03/quick-comparison-go-file-walk-implementations/ UPDATE The below is here for historical reasons, but since 2021 very out of date. See this post https://engineering.kablamo.com.au/posts/2021/quick-comparison-between-go-file-walk-implementations for an updated comparison. Whats the fastest way to get all the names of all files in a directory using Go? I had a feeling that the native walk might not be the fastest way to do it. A quick search showed that several projects claimed to be faster. Since the application I am currently working on needs a high performance scanner I thought I would try the main ones out. https://boyter.org/2018/03/collection-favorite-optimization-posts-articles/ Thu, 08 Mar 2018 22:34:02 +0000 https://boyter.org/2018/03/collection-favorite-optimization-posts-articles/ A collection of my favorite posts to read and re-read about optimizing code to an extreme. Unlikely that I will ever need to go to the extremes that these very talented individuals go to but its nice to learn the techniques. In no particular order. Announcement of Ripgrep a tool for searching code. Brilliantly written with benchmarks and analysis to explain what is happening for each https://blog.burntsushi.net/ripgrep/ the HN comments about it are worth reading as well https://news. https://boyter.org/2018/03/licensechecker-command-line-application-identifies-software-license/ Thu, 01 Mar 2018 07:24:29 +0000 https://boyter.org/2018/03/licensechecker-command-line-application-identifies-software-license/ A simple blog post here to introduce a new command line tool licensechecker (lc), which is similar in purpose to the library Licensee http://ben.balter.com/licensee/ which attempts to identify what software license(s) code is released under. lc itself is dual licensed under the MIT and Unlicense. Licensechecker (lc) was designed to be run either on your command line or using your CI tool of choice and produce either CSV, JSON, SPDX, Summary or CLI tabular or progress output with the option to write results to disk. https://boyter.org/2017/12/simply-encrypt-string-boto3-python-aws-kms/ Mon, 18 Dec 2017 02:54:59 +0000 https://boyter.org/2017/12/simply-encrypt-string-boto3-python-aws-kms/ Another one of those things I need to look up every now and then. Below is a snippet of how to encrypt and decrypt a string using Python and KMS in AWS. The interesting thing is that you don’t need to supply the KMS key alias in the decryption portion. So long as whatever role or key you are using can access the key it should work. For the encryption you can either supply the full ARN of the key or the alias so long as you prefix it with alias/ https://boyter.org/2017/12/searchcode-plexus/ Tue, 05 Dec 2017 01:12:02 +0000 https://boyter.org/2017/12/searchcode-plexus/ Plexus “A combination of interlaced parts; a network.” For a while I have been neglecting searchcode.com while working on searchcode server. This is of course not an ideal situation and as such I have started working on it again. The following is just a brief list of things I am considering, problems, issues etc… So back when I first started working on searchcode I wrote it using PHP. For searchcode next (the current version) I rewrote it using Python and Django. https://boyter.org/2017/10/bbq-dutch-ictstart-delegation/ Tue, 17 Oct 2017 00:22:18 +0000 https://boyter.org/2017/10/bbq-dutch-ictstart-delegation/ So a few weeks back I received an interesting email that at first thought looked like a scam. In short I was being invited to a BBQ at the Consul General’s residence in Sydney to meet an ICT/start up delegation from the Netherlands. Let me straight up say I have no idea why I would receive something like this. I am not dutch, have no dutch ancestry (that I am aware of) and while searchcode. https://boyter.org/2017/09/working-rust/ Sat, 09 Sep 2017 21:59:08 +0000 https://boyter.org/2017/09/working-rust/ For a while I have been wanting to play with a new programming language. Professionally I work with Java, C#, Python and JavaScript and so I went looking for something complimentary to them. My first stop was was Go. I started by implementing a vector space search in it which you can see here https://github.com/boyter/golangvectorspace While I liked the syntax (I still cannot think in Go), the libraries and the performance I realized that Go is close to a hybrid of Java and Python for me. https://boyter.org/2017/06/design-searchcode-server/ Tue, 27 Jun 2017 08:18:05 +0000 https://boyter.org/2017/06/design-searchcode-server/ A very brief update about the progress of searchcode server. Currently I am in the middle of reworking how the index is built and maintained. The idea being I want to add zero downtime index rebuilds which requires a blue/green index strategy. It is still very much in flux but the current design is to merge the indexer and searcher which should allow this to happen. I have been playing around with using an iPad as a production device these days and produced the following document. https://boyter.org/2017/06/home-battery-systems-de-rate-system-capacity/ Wed, 14 Jun 2017 08:06:12 +0000 https://boyter.org/2017/06/home-battery-systems-de-rate-system-capacity/ A quick post. I was looking at the prices to install a home battery system the other day. During the sales process I was informed that for a modular system such as the one provided by Ampetus that if you don’t buy 2 or more batteries that “you may de rate system capacity” Sound’s impressively scary! Slightly worried I asked that it means. Thankfully what it actually means is that instead of getting 3kW delivery using two batteries that one battery matched with the inverter may only deliver 1. https://boyter.org/2017/05/identify-software-licenses-python-vector-space-search-ngram-keywords/ Mon, 08 May 2017 06:54:34 +0000 https://boyter.org/2017/05/identify-software-licenses-python-vector-space-search-ngram-keywords/ EDIT – I have since taken the ideas below improved them and released a command line application you can use to build software license reports https://github.com/boyter/lc/ The below is mostly a log of my thought process while building out some functionality that I wanted to add into searchcode server. I kept a record of progress and thoughts while doing this in the hopes that I get some sort of useful blog post out of it. https://boyter.org/2017/03/golang-solution-faster-equivalent-java-solution/ Thu, 30 Mar 2017 22:52:01 +0000 https://boyter.org/2017/03/golang-solution-faster-equivalent-java-solution/ At work there is a tradition of a Friday quiz being posted by the winner of the previous week. I missed out on the most recent one due to having to duck off early to do my tax but the problem was rather an interesting one. The challange itself is not as simple as you would initally think and taken from a 2015 IBM Ponder This https://www.research.ibm.com/haifa/ponderthis/challenges/May2015.html Three people are playing the following betting game. https://boyter.org/2017/02/setup-concourseci-2-6-0-nginx-signed-certificates-ubuntu-16-04/ Sun, 12 Feb 2017 21:56:58 +0000 https://boyter.org/2017/02/setup-concourseci-2-6-0-nginx-signed-certificates-ubuntu-16-04/ Concourse CI is a very nice continuous integration server. However for installs there are a few gotcha’s you need to keep in mind. Mostly these relate to how TLS/SSL works. The first is that while it is possible to run concourse inside Docker I found this to cause a lot of issues with workers dying and not recovering. I would suggest installing the binarys on bare machines. When I moved from a docker cluser using Amazon’s ECS to a single t2. https://boyter.org/2017/01/repository-overview-searchcode-server/ Mon, 30 Jan 2017 07:34:35 +0000 https://boyter.org/2017/01/repository-overview-searchcode-server/ One feature that I have wanted for a long time in searchcode server was a page which would give an overview of a repository. I wanted the overview to give a very high look at the languages used, the total number of files, estimated cost and who would be the best people to talk to. One thing that occurred to me when I started work was that it would be nice to calculate a bus factor for the repository as well. https://boyter.org/2016/11/sphinx-real-time-index-distribute-hidden-gotcha/ Fri, 11 Nov 2016 09:34:42 +0000 https://boyter.org/2016/11/sphinx-real-time-index-distribute-hidden-gotcha/ I have been working on real time indexes with Sphinx recently for the next version of searchcode.com and ran into a few things that were either difficult to search for or just not covered anywhere. The first is how to implement a distributed search using real time indexes. It’s actually done the same way you would normally create an index. Say you had a single server with 4 index shards on it and you wanted to run queries against it. https://boyter.org/2016/10/explaining-varnishhist/ Wed, 26 Oct 2016 23:56:01 +0000 https://boyter.org/2016/10/explaining-varnishhist/ The varnishhist tool is one of the most underused varnish tools that come with your standard varnish install. Probably because of how it appears at first glance. In short, you want as many | symbols as possible and you want everything far toward the left hand side. The closer to the left the faster the responses are regardless if they are cached or not. The more | symbols then more items were served from cache. https://boyter.org/2016/10/mysql-dump-impacting-queries/ Wed, 26 Oct 2016 01:11:01 +0000 https://boyter.org/2016/10/mysql-dump-impacting-queries/ Posted more for my personal use (I have to look it up every time) but here is how to run a mysqldump without impacting performance on the box. It sets the ionice and nice values to be as low as possible (but still run) and uses a single transaction and ups the max packet size for MySQL. ionice -c2 -n7 nice -n19 mysqldump -u root -p DATABASE --single-transaction --max_allowed_packet=512M > FILENAME https://boyter.org/2016/09/companies-recruiting/ Thu, 08 Sep 2016 22:29:19 +0000 https://boyter.org/2016/09/companies-recruiting/ I am writing this on behalf of all developers/engineers out there. Please stop with the take home coding challenge questions. Really. Just stop it. They are a lazy and frankly an unprofessional way of sorting the wheat from the chaff. Before closing your browser in disgust hear me out on this one and hopefully I can convince you of the error of your ways. There has become an alarming trend these days of companies during the hiring process to issue lengthy coding challenges in order to prove that the individual they are hiring knows their stuff. https://boyter.org/2016/08/gpl-time-bomb-interesting-approach-foss-licensing/ Mon, 29 Aug 2016 01:47:39 +0000 https://boyter.org/2016/08/gpl-time-bomb-interesting-approach-foss-licensing/ UPDATES Following some feedback I am going to rename my usage of “Time-Bomb” due to potential negative connotation on the words. I am going to call it “Eventually Open”. Also a few other things need mentioning. I am not looking for code submissions back into the source at this time. This was a move to show that there are no back-doors in the code sending source code back to a master server. https://boyter.org/2016/08/searchcode-server-fair-source/ Wed, 24 Aug 2016 23:38:33 +0000 https://boyter.org/2016/08/searchcode-server-fair-source/ A very quick blog today. I have released searchcode server under the fair source licence. This means that as of a few days ago you can view the source, change it modify it and run it as you see fit so long as you have less than 5 users. The source is hosted on github (I may move this to GitLab sometime in the future) and you can view it here. https://boyter.org/2016/08/worst-individual-worked/ Tue, 23 Aug 2016 02:55:20 +0000 https://boyter.org/2016/08/worst-individual-worked/ Taken from a comment I posted on HN in a thread about a Soccer Con Man. Not actually a programmer. The guy was hired to be a project manager. After joining things were as expected but after a few weeks we noticed that he was rarely around after lunch and never around after lunch on a Friday. We would email him at those times deliberately to catch him out and I recall starting to put sticky notes on his laptop “Came to see you a X time”. https://boyter.org/2016/08/types-testing-software-engineering/ Wed, 17 Aug 2016 22:42:37 +0000 https://boyter.org/2016/08/types-testing-software-engineering/ There are many different types of testing which exist in software engineering. They should not be confused with the test levels, unit testing, integration testing, component interface testing, and system testing. However the different test levels may be used by each type as a way of checking for software quality. The following are all different types of tests in software engineering. A/B A/B testing is testing the comparison of two outputs where a single unit has changed. https://boyter.org/2016/08/syncing-stashbitbucket-searchcode-server/ Thu, 04 Aug 2016 05:14:19 +0000 https://boyter.org/2016/08/syncing-stashbitbucket-searchcode-server/ Recently it came up to perform a slight integration piece between a on premises Stash/BitBucket install and a searchcode server install. Thankfully both have an API and very thankfully there is a nice Python library for talking to Stash/BitBucket. Below is the code used. It pulls out all of the repositories from every project, checks if it exists in searchcode and if not adds it as a repository to be indexed. https://boyter.org/2016/08/python-fabric-file-host-string/ Mon, 01 Aug 2016 22:52:39 +0000 https://boyter.org/2016/08/python-fabric-file-host-string/ When using fabric for deployments you will sometimes want check an existing file for the presence of a value before applying an update. A common example I run into is checking if an apt-source has already been added before adding it again. This is a little clunky in fabric, but thankfully you can write a simple helper which takes case of it for you. def _get_remote(fileloc): '''Pulls back a file contents from connection as string''' from StringIO import StringIO fd = StringIO() get(fileloc, fd) content = fd. https://boyter.org/2016/07/searchcode-com-architecture-migration-3-0/ Thu, 28 Jul 2016 01:09:56 +0000 https://boyter.org/2016/07/searchcode-com-architecture-migration-3-0/ 27th July 2016 at about 9:30pm my local time (GMT +10) I updated the A records for searchcode’s nameservers to point at a new stack that has been several months in the making. As with most posts of this sort of nature a quick recap of where things were and where they are now. The previous searchcode stack consisted of two dedicated servers hosted by Hetzner. I have previously discussed this about two years ago when discussing searchcode next. https://boyter.org/2016/07/hide-methods-fabric-task-listing/ Thu, 21 Jul 2016 22:34:49 +0000 https://boyter.org/2016/07/hide-methods-fabric-task-listing/ Occasionally you may want to hide a method from appearing inside the fabric listing of available tasks. Usually its some sort of helper method you have created that is shared by multiple tasks. So how to hide it? Simply prefix with _ For example, def _apt_get(packages): '''Makes installing packages easier''' sudo('apt-get update') sudo('apt-get -y --force-yes install %s' % packages) When listing the fabric tasks this method will no longer appear in the results. https://boyter.org/2016/07/python-fabric-show-list-tasks/ Wed, 20 Jul 2016 22:30:51 +0000 https://boyter.org/2016/07/python-fabric-show-list-tasks/ Showing or displaying the available tasks inside a fabric fabfile is one of those things that almost everyone wants to do at some point and usually works out you can just request a task you know will not exist (usually found through a typo). However there is a way to list them built into fabric itself. The below are all methods which can be used to display the currently defined tasks. https://boyter.org/2016/07/set-ubuntu-linux-swapfile-python-fabric/ Tue, 19 Jul 2016 22:53:52 +0000 https://boyter.org/2016/07/set-ubuntu-linux-swapfile-python-fabric/ Annoyingly most cloud providers have an irritating habit of not adding any swap memory to any instance you spin up. Probably because if they added swap to the instance the disk size would appear to be smaller then it is or if they had a dedicated swap partition they would have to bear the cost or again use some of your disk space. Thankfully adding swap to your Ubuntu linux instance is fairly easy. https://boyter.org/2016/07/python-fabric-set-host-list-runtime/ Mon, 18 Jul 2016 22:40:13 +0000 https://boyter.org/2016/07/python-fabric-set-host-list-runtime/ With the advent of cloud computing where you spin up and tear down servers at will it becomes extremely useful to pick the hosts you want fabric to run on at runtime rather then through the usual env.hosts setting. This allows you to query your servers through your cloud providers API without having to maintain a list. This can be a more powerful and flexible technique then using roles and in a devops world can save you a lot of time. https://boyter.org/2016/07/chaos-testing-engineering/ Sat, 02 Jul 2016 07:48:29 +0000 https://boyter.org/2016/07/chaos-testing-engineering/ A blog post by the excellent technical people at Netflix about Chaos Engineering and further posts about the subject by Microsoft in Azure Search prompted me to ask the question, What is chaos engineering and how can chaos testing be applied to help me? What is Chaos Testing? First coined by the afore-mentioned Netflix blog post, chaos engineering takes the approach that regardless how encompassing your test suite is, once your code is running on enough machines and reaches enough complexity errors are going to happen. https://boyter.org/2016/04/collection-orly-book-covers/ Fri, 22 Apr 2016 00:48:41 +0000 https://boyter.org/2016/04/collection-orly-book-covers/ A quick collection of all the fake Orly book covers I find around on the internet. I am seriously considering writing content for the “Expert Excuses for Not Writing Unit Tests” one. EDIT - I actually did write content and publish it as a book! Expert Excuses for Not Writing Unit Tests https://boyter.org/2016/03/searchcode-server-released/ Thu, 31 Mar 2016 21:19:33 +0000 https://boyter.org/2016/03/searchcode-server-released/ searchcode server the downloadable self hosted version of searchcode.com is now available. A large amount of work went into the release with a variety of improvements based on feedback from the general beta releases. searchcode server has a number of advantages over searchcode.com that will eventually be back-ported in. The full list of things to check out is included below, New Single Page Application UI for smooth search experience Ability to split on terms so a search for “url signer” will match “UrlSigner” Massively improved performance 3x in the worst case and 20x in the best Configurable through UI and configuration Spelling suggestion that learns from your code A few things of note, https://boyter.org/2015/12/searchcode-server/ Tue, 29 Dec 2015 02:17:42 +0000 https://boyter.org/2015/12/searchcode-server/ A month or so ago I started collection emails on searchcode.com to determine if there was enough interest in a downloadable version of searchcode. The results were overwhelmingly positive. The email list grew far beyond what I would have expected, and this was in the first month. As such I have been working in this downloadable version of searchcode which will probably be called searchcode server. Progress has been reasonably straight forward consider that searchcode. https://boyter.org/2015/12/pi-hole-ubuntu-14-04/ Sat, 26 Dec 2015 02:24:32 +0000 https://boyter.org/2015/12/pi-hole-ubuntu-14-04/ Because of the fact that I personally work for an ad supported company and that searchcode.com is currently supported via third party advertising I tend to keep an eye on the state of ad blockers on the web. Most people probably know about adblockplus and other browser extensions however there are other ways to block ad’s on ones network. One that I had previously read about was setting up your own Bind9 server on a server and adding custom rules to block them at a DNS level. https://boyter.org/2015/10/searchcode-local/ Fri, 30 Oct 2015 07:56:37 +0000 https://boyter.org/2015/10/searchcode-local/ I am going to copy the searchcode pitch itself below quickly before explaining it a bit further. “searchcode offers powerful code search over billions of lines of open source code. Imagine what it could do with your private repositories. There have been requests to offer a downloadable version of searchcode. Given enough interest a downloadable hostable version of searchcode will be offered. Register your email below to register your interest. https://boyter.org/2015/09/search/ Wed, 02 Sep 2015 22:25:18 +0000 https://boyter.org/2015/09/search/ A very fast update. At the request of the excellent Lars Brinkhoff via GitHub I have added in the language Forth to be one of the supported languages inside searchcode. An example search which shows this working would be the following https://searchcode.com/?q=forth&loc=0&loc2=10000&lan=181 I had to solve a number of interesting problems inside searchcode to support this change. For pragmatic reasons the way searchcode identifies what language any piece of code is written in is to run it though CLOC (Count Lines Of Code). https://boyter.org/2015/09/exporting-documents-knowledgetree-3-7-0-2/ Tue, 01 Sep 2015 22:13:04 +0000 https://boyter.org/2015/09/exporting-documents-knowledgetree-3-7-0-2/ I was recently tasked with exporting a large collection of documents from KnowledgeTree (KT) for a client. The collection was too large to use the download all functionality and too wide to attempt to export each folder individually. I had played around with the WebDav connection that KT provides but it either didn’t work or was designed deliberately to not allow exporting of the documents. I looked at where the documents were stored on disk but KT stores them as numbered files in numbered directories sans extension or folder information. https://boyter.org/2015/08/decoding-captchas-handbook/ Tue, 25 Aug 2015 22:49:31 +0000 https://boyter.org/2015/08/decoding-captchas-handbook/ Some time ago I wrote an article about Decoding CAPTCHA’s which has become what appears to be the first resource most people encounter when searching for information in the decoding CAPTCHA space. I had continued to write about CAPTCHA’s over the years with posts scattered around the web. A while ago I started to consolidate all of my content on this blog and realised that I had considerably more CAPTCHA related articles then I thought. https://boyter.org/2015/08/running-hours-ruby-tests-minutes/ Tue, 18 Aug 2015 08:18:47 +0000 https://boyter.org/2015/08/running-hours-ruby-tests-minutes/ Recently the very cool hard working developers working on Stripe released a post about how they modified their build/test pipeline to reduce their test suite runtime from 3 hours to about 3 minutes. The article is very much worth reading, as is the discussions that have come around it including those on Hacker News. A few key takeaways, For dynamic languages such as Ruby or Python consider forking to run tests in parallel https://boyter.org/2015/08/culture-quality/ Mon, 10 Aug 2015 08:17:33 +0000 https://boyter.org/2015/08/culture-quality/ The best working environment I had the pleasure to work in had a strong emphasis on testing and software quality in general. Product teams were encouraged to spend extra time ensuring that everything worked over shipping before it was ready. The transformation it went through was incredible. Having come from a culture very much wild west through to where it was. An example of the advantages this brought was that before adopting this culture website launches were a traumatic event. https://boyter.org/2015/08/c-language-google-post/ Wed, 05 Aug 2015 22:37:38 +0000 https://boyter.org/2015/08/c-language-google-post/ The more I use C# as a language for writing things the more I am convinced that its approach really is the best language approach out there. The unit test support is excellent which allows development speed to be just as fast as any dynamic language (Python, PHP, Perl). The static typing catches so many issues before you get to runtime and allows sweeping changes without breaking things. Unlike Java it has the var keyword (saves time and improves readability) and so many more useful functions which yes you can replicate but are just built in and work correctly. https://boyter.org/2015/07/ab-testing/ Mon, 27 Jul 2015 08:11:59 +0000 https://boyter.org/2015/07/ab-testing/ A/B testing is testing the comparison of two outputs where a single unit has changed. It is commonly used when when trying to increase conversion rates for online websites. Also known as split testing an example would be trying to increase user clicks on a specific button on a website. You may have a theory that red buttons work better then green. You would try out both against real users and see which one performs better. https://boyter.org/2015/07/ways-avoid-control-flaky-tests/ Tue, 21 Jul 2015 08:09:42 +0000 https://boyter.org/2015/07/ways-avoid-control-flaky-tests/ Having a reliable test suite should always be the goal in software development. After all if you can’t trust the tests then why bother running them at all? This is especially important in a shared coding environment and when running through Continuous Integration (CI). 1 - Test in Isolation It may seem obvious but writing focused tests which do a single thing is one of the most effective ways to avoid them being flaky. https://boyter.org/2015/07/software-bugs/ Mon, 20 Jul 2015 08:06:50 +0000 https://boyter.org/2015/07/software-bugs/ “Why does all software contain bugs?” this was a question recently asked of me. My response at the time was because all software is not perfect, but is this true? Lets take a very simple example. public class Hello { public static void main(String[] args) { System.out.println("Hello World!"); } } The above example is arguably the simplest program that can be written using Java. It also happens to be the first program usually written by any Java programmer. https://boyter.org/2015/07/searchcode-path-profitability/ Fri, 17 Jul 2015 08:01:15 +0000 https://boyter.org/2015/07/searchcode-path-profitability/ One of the things that has always bothered me about searchcode.com was that it never generated any money. Not a huge problem in itself as a side project, but the costs to run it are not insignificant due to the server requirements. I had looked into soliciting donations but I considered this highly unlikely to produce enough revenue to cover costs considering that sites such as gwern.net was unable to make enough to cover even basic costs through patreon (although since a recent HN post this has jumped from around $20 a month to over $150). https://boyter.org/2015/07/benefit-testing-developers-managers-business/ Thu, 16 Jul 2015 08:05:30 +0000 https://boyter.org/2015/07/benefit-testing-developers-managers-business/ “Fixing regression bugs is analogous to digging a hole only to find the next day it has been filled in and having to dig it out again” Ask any manager, developer or tester working on software without tests what the main pain points are. Nearly all the time the main one mentioned is dealing with regressions. This is bugs that were fixed a year ago which returned. Regression bugs cost the software industry billions of dollars a year. https://boyter.org/2015/07/aws-ec2-instance-types-test-agents/ Fri, 10 Jul 2015 08:03:10 +0000 https://boyter.org/2015/07/aws-ec2-instance-types-test-agents/ When you are running test agents on AWS knowing what instance type to run as test agents (for TeamCity or otherwise) can involve a lot of trial and error. Not only can there be great savings to be made by picking the correct instance type you can speed up your builds and get test feedback back faster which can be far more valuable the cost of a few additional cents an hour. https://boyter.org/2015/07/issues-googles-bug-prediction-algorithm/ Wed, 08 Jul 2015 07:58:57 +0000 https://boyter.org/2015/07/issues-googles-bug-prediction-algorithm/ December 2011 the Google Engineering team published a blog post about bug prediction at Google. The topic caused quite a lot of discussion at the time over the internet on forums such as Hacker News and the Reddit programming sub-reddit. How bug prediction works In a nutshell the prediction works by ranking files against checking the file commit history and seeing how many changes have been flagged as bug fixes. Of course this means that code which was previously buggy will still appear in the list. https://boyter.org/2015/07/usability-testing/ Mon, 06 Jul 2015 07:57:30 +0000 https://boyter.org/2015/07/usability-testing/ Usability tests are manual tests used to check that the user interface is understandable. The focus of the tests are to ensure that product meets its intended purpose. These sort of tests can be subjective and are usually impossible to automate. It is important to differentiate usability testing from simply showing an interface to someone and asking them “Do you understand how this?”. It is usually done by creating a scenario such as “Can you find and add this song to a new playlist” and observing the steps that the user takes to perform the task. https://boyter.org/2015/07/mutation-testing-test-tests/ Sun, 05 Jul 2015 07:56:03 +0000 https://boyter.org/2015/07/mutation-testing-test-tests/ Mutation testing is a technique used to verify that tests are providing value. Mutation testing involves modifying the given program in small ways. These could include changing boolean checks such as if a condition is True to being False. A mutated version of code is known as a mutant. For each mutant a the test suite is run against it. The tests when run over the mutant version should have a percentage of failure. https://boyter.org/2015/07/responsible-software-quality/ Sat, 04 Jul 2015 07:54:08 +0000 https://boyter.org/2015/07/responsible-software-quality/ In the beginning of my software development career I was interviewing for an intern position at Microsoft. I never did get the job but one think out of that interview process really stuck with. The second interviewer after the usual getting to know you chat aded me the following question. “On any given software project we have developers, software testers / quality assurance and managers involved. Who is responsible for the quality of the software? https://boyter.org/talks/ Sat, 04 Jul 2015 06:12:08 +0000 https://boyter.org/talks/ A few of the talks I have given and the slide desks are included below. Python @ searchcode.com sypy_searchcode.key (keynote) sypy_searchcode (powerpoint) Decoding CAPTCHA’s for Fun and Profit DecodingCaptchas Reveal.js (futher details can be found http://www.boyter.org/2015/03/decoding-captchas-presentation/) How AWS and Go Transformed a Public Broadcaster https://boyter.org/static/aws-go-archive-presso/ Sloc Cloc and Code - Can a crusty Go program outperform a well written Rust Project? https://boyter.org/static/gophercon-syd-presentation/ https://www.youtube.com/watch?v=jd-sjoy3GZo DataEngBytes 2023 - Processing 40 TB of code from ~10 million projects with a dedicated server and Go for $100 https://boyter. https://boyter.org/2015/07/story-hubris-integration-tests/ Fri, 03 Jul 2015 07:51:07 +0000 https://boyter.org/2015/07/story-hubris-integration-tests/ Philip Dormer Stanhope, 4th Earl of Chesterfield (pictured) managed to embarrass me in front of my peers once. Sort of. In truth it was my hubris that caused the incident. Here is how it happened and what I learnt through the process. In the summer of 2010 I was tasked with developing a new application where I worked. The requirement was fairly simple “We need a web application to upload a CSV”. https://boyter.org/2015/07/searchcode-com-unit-integration-tested/ Wed, 01 Jul 2015 07:46:18 +0000 https://boyter.org/2015/07/searchcode-com-unit-integration-tested/ searchcode is a source code and documentation search engine. It allows users to is search over 20 billions lines of code and millions of API documentation. It at the time of writing gets over 300,000 unique visitors a month processes many millions of API requests. What follows is how I am testing searchcode, the issues I hit and where I think I am getting the most value. As big believer in testing as a methodology of improving code quality, one of my goals when rewriting searchcode was to ensure that it had a comprehensive test suite. https://boyter.org/2015/06/flaky-tests/ Tue, 30 Jun 2015 22:50:04 +0000 https://boyter.org/2015/06/flaky-tests/ A test is considered flaky or flakey if it if fails occasionally. Generally flaky tests are considered to be a bad thing and should be modified to ensure they work correctly every time. This is because a test that is not trustworthy will be ignored even when indicating real failure. There are many situations that can cause to become flaky. Integration and acceptance tests are generally the tests in your test suite most likely to become flaky. https://boyter.org/2015/06/clean-testable-repository-data-access-sharp/ Tue, 30 Jun 2015 22:44:13 +0000 https://boyter.org/2015/06/clean-testable-repository-data-access-sharp/ Below is an implementation of an extremely clean data access pattern possible using C# and Entity Framework. It saves you the effort of mocking the database context as the code you end up writing is so simple it is all compile time checked. The advantages of this are firstly that everything is very easy to test as you can perform all joins in your service layer with mocks of the repository. https://boyter.org/2015/06/unsung-benefits-software-testing/ Mon, 29 Jun 2015 22:35:06 +0000 https://boyter.org/2015/06/unsung-benefits-software-testing/ One benefit that is generally not talked about when discussing testing is the following. The feeling of productivity because you are writing lots of code. Think about that for a moment. Ask any developer who wants to develop why they became a developer. One of the first things that comes up is “I enjoy writing code”. This is one of the things that I personally enjoy doing. Writing code, any code especially when its solving my current problem makes me feel productive. https://boyter.org/2015/06/testing-software-engineering/ Sun, 28 Jun 2015 22:38:23 +0000 https://boyter.org/2015/06/testing-software-engineering/ Testing is software engineering is a method of providing information about code quality when developing a piece of software. The intent of writing and running tests is the enforce good software design and identify software bugs and defects. These defects can include specification/requirement errors as well as developer mistakes. The general aim of software testing is the ensure that software meets the following goals, Meets its requirements gathered before its design and implementation https://boyter.org/2015/06/grouping-tests-unitintegration-fastslow-tests/ Sun, 28 Jun 2015 22:36:47 +0000 https://boyter.org/2015/06/grouping-tests-unitintegration-fastslow-tests/ There is a great deal of argument in the testing community over how to label tests. One camp likes to label tests using levels such that unit tests are in one group, integration in another and so forth. The other likes to label them based on how long they take to run ignoring what level they are in. Fast tests are those that run in milliseconds while slow take longer then this. https://boyter.org/2015/06/sanity-testing/ Sun, 28 Jun 2015 22:13:41 +0000 https://boyter.org/2015/06/sanity-testing/ Software sanity tests are closely associated with smoke tests. They attempt to determine if is reasonable to continue with testing a given piece of software. The objective is not to test all functionality, but to determine if there is value in doing so. You can consider it a “Should I continue to test this?” check. Sanity tests differ from smoke tests as they exist to check if new functionality has been met and existing bugs have been resolved. https://boyter.org/2015/04/filetree-listing/ Tue, 14 Apr 2015 21:32:31 +0000 https://boyter.org/2015/04/filetree-listing/ Just a quick update to searchcode. A few small tweaks here and there, but the largest is that there is now a file tree listing option which will show the file tree for any project. An example would be going to this file and then clicking the “View File Tree” button on the top left. An example screenshot of the result of this is included below. https://boyter.org/2015/03/updates-searchcode-com/ Wed, 18 Mar 2015 22:11:33 +0000 https://boyter.org/2015/03/updates-searchcode-com/ Just a quick post to list some updates to searchcode.com The first is a slight modification to the home page. A while ago I received an email from the excellent Christian Moore who provided some mock-ups of how he felt it should look. I loved the designs, but was busy working on other issues. Thankfully however in the last week or so I found the time to implement his ideas and the result is far more professional to me. https://boyter.org/2015/03/decoding-captchas-presentation/ Sun, 08 Mar 2015 20:41:13 +0000 https://boyter.org/2015/03/decoding-captchas-presentation/ A few days ago there was a lack of speakers for #SyPy which is the Sydney Python meet-up held most months and sponsored by Atlassian. I had previously put my hand up to help out if this situation ever came up and was mostly ready with a presentation about Decoding Captchas. I did not expect it to be so full that people were standing (largest crowd I had ever seen there). https://boyter.org/2015/02/c-xml-cleaner-regex/ Thu, 19 Feb 2015 22:41:47 +0000 https://boyter.org/2015/02/c-xml-cleaner-regex/ One of the most annoying things I deal with is XML documents with invalid characters inside them. Usually caused by copy pasting from MS Word it ends up with invisible characters that you cannot easily find and cause XML parsers to choke. I have encountered this problem enough that I thought a quick blog post would be worth the effort. As such here mostly for my own reference is a regular expression for C# . https://boyter.org/2015/02/regular-expressions-fast/ Sun, 15 Feb 2015 21:36:40 +0000 https://boyter.org/2015/02/regular-expressions-fast/ TL/DR: Regular expressions are fast, until they aren’t. How I got a 20x performance by switching to string functions. With the new version of searchcode.com one of the main things I wanted to address was performance. The previous version had all sorts of performance issues which were not limited to the usual suspects such as the database or search index. When developing the new version one of the tasks listed in my queue was to profile search operations for anything slowing things down. https://boyter.org/2015/01/rebound-project/ Fri, 30 Jan 2015 22:49:55 +0000 https://boyter.org/2015/01/rebound-project/ As I mentioned in the previous entry I had started work on a new project I called portfold. Built and released without fanfare I have quietly killed it before even the month is out. Why? I realise now that it was a rebound project similar to a rebound relationship. I had been getting a little down on searchcode.com and wanted to branch out to some new technology. Once done though the itch was scratched and now I am back to working on searchcode again. https://boyter.org/2014/12/portfold-topic-research-software/ Sun, 07 Dec 2014 22:39:44 +0000 https://boyter.org/2014/12/portfold-topic-research-software/ Every few years I have a habit of starting a new project. The goal always being to scratch my own itch and learn some new technology in the process. While I am still working on searchcode.com I really wanted to play with what I had learnt there and apply it to something new. You can view it at portfold.com Recently I have been taking an interest in various topics such as “Oil Gas Pipeline Failure Rates” and “Hydroelectric Dam Environmental Impacts” (both generated using Portfold). https://boyter.org/2014/10/searchcode-com-100-free-software/ Fri, 10 Oct 2014 07:14:08 +0000 https://boyter.org/2014/10/searchcode-com-100-free-software/ The recent surge in attention to searchcode from the Windows 9/10 naming fiasco resulted in a lot of questions being raised about searchcode’s policy about free software (open source). While the policy is laid out on the about page some have raised the issue of the ethics about using such a website which is not 100% free (as in freedom). For the purposes of the rest of this post “free software” is going to refer to software defined as not infringing freedom rather then free as in beer. https://boyter.org/2014/08/interesting-code-comment/ Mon, 11 Aug 2014 22:58:54 +0000 https://boyter.org/2014/08/interesting-code-comment/ Found the following comment in some code I had modified a few years ago. Just to set this up, its an existing application I had no hand in creating, and is a totally atrocity of 180,000 lines of untested code (and pretty much un-testable) which through the abuse of extension methods lives in a single class spread out across multiple files. https://boyter.org/2014/07/feedback-loop/ Mon, 28 Jul 2014 23:37:36 +0000 https://boyter.org/2014/07/feedback-loop/ About a month ago searchcode.com managed to sit on the front page of Hacker News (HN) for most of a day and produced a lot of useful feedback for me to act on. You can read the full details here searchcode: A source code search engine Between the HN feedback, some I received via tweets and from republished articles I got a list of things I needed to work on. https://boyter.org/2014/07/special-ddg/ Sat, 19 Jul 2014 02:37:05 +0000 https://boyter.org/2014/07/special-ddg/ Since I am still bringing all my content together I thought I would pull in this post from Quora asking what is special about DuckDuckGo. Privacy enabled by default. This certainly helped get traction when the NSA security revelations came around. DDG is not the only privacy conscious search engine but certainly one that pushes it as a feature more then others. See https://duckduckgo.com/privacy !bang syntax. Remember back in the early days of Google they had a “Try this search on” and a list of search engines? https://boyter.org/2014/07/yourstory-interview/ Tue, 08 Jul 2014 22:58:58 +0000 https://boyter.org/2014/07/yourstory-interview/ Aditya Dwivedi recently did an interview with me about searchcode. You can view it here over on yourstory.com and it covers how the idea came about, some of the issues and where I think the future lies for the project. Going to include an extract below in case bit rot sets in. How this Aussie coder is making life easy for techies around the world with searchcode You are busy building your product and then someone comes and tells you that what you’ve been doing has already been done by someone. https://boyter.org/2014/06/sphinx-searchcode/ Fri, 20 Jun 2014 00:02:35 +0000 https://boyter.org/2014/06/sphinx-searchcode/ There is a rather nice blog post on the Sphinx Search blog about how searchcode uses sphinx. Since I wrote it I thought I would include a slight edited for clarity version below. You can read the original here. I make it no secret that the indexer that powers searchcode is Sphinx Search which for those who do not know is a stand alone indexing and searching engine similar to Solr. https://boyter.org/2014/06/estimating-sphinx-search-ram-requirements/ Thu, 19 Jun 2014 10:57:03 +0000 https://boyter.org/2014/06/estimating-sphinx-search-ram-requirements/ If you run Sphinx Search you may want to estimate the amount of RAM that it requires in order to per-cache. This can be done by looking at the size of the spa and spi files on disk. For any Linux system you can run the following command in the directory where your sphinx index(s) are located. ls -la /SPHINXINDEX/|egrep "spa|spi"|awk '{ SUM += $5 } END { print SUM/1024/1024/1024 }' This will print out the number of gigabytes required to store the sphinx index in RAM and is useful for guessing when you need to either upgrade the machine or scale out. https://boyter.org/2014/06/searchcode/ Mon, 16 Jun 2014 23:16:45 +0000 https://boyter.org/2014/06/searchcode/ There seems to be a general trend with calling the new release of your search engine next (see Iconfinder and DuckDuckGo), and so I am happy to announce and write about searchcode next. As with many project searchcode has some very humble beginnings. It originally started out as a “I need to do something” side project originally just indexing programming documentation. Time passed and the idea eventually evolved into a search engine for all programming documentation, and then with Google Code search being shut down a code search engine as well. https://boyter.org/2014/06/chatbot-nonsense/ Mon, 09 Jun 2014 02:52:52 +0000 https://boyter.org/2014/06/chatbot-nonsense/ There has been a lot of coverage recently about a chat-bot “Eugene Goostman” passing a variance of the Turing test by convincing 30% of the testers that it was indeed a human by posing as Ukrainian 13 year old boy (to make misspellings and grammar mistakes forgivable I suppose). Naturally I had to give it a try and frankly I can’t see how something like this could convince anyone that its a living human being. https://boyter.org/2014/05/unique-guid/ Wed, 28 May 2014 10:43:13 +0000 https://boyter.org/2014/05/unique-guid/ I have been doing a lot of work with the Sitecore CMS recently. Once of the things you quickly learn is how it relies on GUID’s for pretty much everything. This means of course when you start testing and need to supply GUID’s into your tests that you end up with lots of GUIDs that look like the following sprinkled through your code {11111111-1111-1111-1111-111111111111} Today I remarked that we should be using things like “deadbeef” for the first part of the GUID with a colleague. https://boyter.org/2014/05/implementing-c-linq-distinct-custom-object-list/ Wed, 07 May 2014 23:11:49 +0000 https://boyter.org/2014/05/implementing-c-linq-distinct-custom-object-list/ Ever wanted to implement a distinct over a custom object list in C# before? You quickly discover that it fails to work. Sadly there is a lack of decent documentation about this and a lot of FUD. Since I lost a bit of time hopefully this blog post can be picked up as the answer. Thankfully its not as difficult as you would image. Assuming you have a simple custom object which contains an Id, and you want to use that Id to get a distinct list all you need to do is add the following to the object. https://boyter.org/2014/04/installing-phindex/ Sun, 13 Apr 2014 09:05:34 +0000 https://boyter.org/2014/04/installing-phindex/ This is a follow on piece to my 5 part series about writing a search engine from scratch in PHP which you can read at http://www.boyter.org/2013/01/code-for-a-search-engine-in-php-part-1/ I get a lot of email requests asking how to setup Phindex on a new machine and start indexing the web. Since the article and code was written aimed at someone with a degree of knowledge of PHP this is somewhat understandable. What follows is how to set things up and start crawling and indexing from scratch. https://boyter.org/2014/04/interview-snippets/ Wed, 02 Apr 2014 08:59:36 +0000 https://boyter.org/2014/04/interview-snippets/ Since I wrote the code to these snippets I thought I may as well add them here in case I ever need them again or want to review them. As the other interview ones they are the answers to a question I was asked, slightly modified to protect the innocent. These ones are written in Python. Q. Write a function to reverse each word in a string. def reverse_each_word(words): ''' Reverse each word in a string ''' return " ". https://boyter.org/2014/04/day-interview/ Tue, 01 Apr 2014 21:43:58 +0000 https://boyter.org/2014/04/day-interview/ Another day another interview. I actually have been getting some good results from them so far. In particular the last two I have been on. I will discuss them briefly. The first had an interesting coding test. Rather then asking me to solve Fizzbuzz or implement a depth first algorithm over a binary tree (seriously, I have been programming for 10 years and never needed to do that. I can, but its something I did in uni and not really applicable to anything I have done since then). https://boyter.org/2014/03/searchcode-screenshot/ Wed, 26 Mar 2014 03:55:54 +0000 https://boyter.org/2014/03/searchcode-screenshot/ Since I have been working on searchcode for a while and its getting close to being ready for release (a few weeks away at this point I predict) I thought I would post a teaser screenshot. The below shows how it looks for a sample search. The design is far cleaner then what is currently online which is a big win as the current design of searchcode is seriously ugly. https://boyter.org/2014/02/sample-coding-test/ Fri, 28 Feb 2014 02:55:38 +0000 https://boyter.org/2014/02/sample-coding-test/ Being in the job market again I been doing quite a few tests. Since I have already put in the effort to a test without result I thought I would post it here. The test involved producing output from a supplied CSV input file which contained insurance claims. Something about taking the input and using it to predict future claims. Please forgive my explanation as I am not a financial expert. https://boyter.org/2014/02/bitcoin-clones-network/ Fri, 21 Feb 2014 03:00:08 +0000 https://boyter.org/2014/02/bitcoin-clones-network/ Another comment I posted over on the TechZing Podcast. It was addressing Justin’s comment about bitcoin clones using the same “network” which is true, in that they share the same protocol but each have their own blockchain. Each of the “bitcoin” clones are actually their own network. As far as I am aware they have no communication between each network in any form. Its also why each one’s blockchain is so different in size. https://boyter.org/2014/02/python-pep8-git-commit-check/ Thu, 13 Feb 2014 05:34:17 +0000 https://boyter.org/2014/02/python-pep8-git-commit-check/ Without adding a git commit hook I wanted to be able to check if my Python code conformed to pep8 standards before committing anything. Since I found the command reasonably useful I thought I would post it here. git status -s -u | grep '\.py$' | awk '{split($0,a," "); print a[2]}' | xargs pep8 Just run the above in your projects directory. It’s fairly simple but quite effective at ensuring your Python code becomes cleaner ever time you commit to the repository. https://boyter.org/2014/02/zombie-apocalypse/ Wed, 12 Feb 2014 01:00:32 +0000 https://boyter.org/2014/02/zombie-apocalypse/ This piece of content is taken from a comment I left on the TechZing podcast blog. I should note I have not even begun to explore issues such as what happens to a zombie in extreme heat or cold. Of course much of the below can be disregarded if the zombie virus is airborne, but this assumes the standard zombie canon of being spread through bites. My take on the zombie apocalypse was always that it could never happen. https://boyter.org/2014/02/searchcode-logo/ Mon, 10 Feb 2014 22:01:31 +0000 https://boyter.org/2014/02/searchcode-logo/ Just a quick post to show off the new searchcode.com logo. I have been working on a new version of the site for a few weeks now and want to get something out there for people to look at. Design wise its not done yet, but the logo is. There it is in all its glory. The new design has a similar look and I should be able to start talking about and showing it off soon. https://boyter.org/2014/02/worst-program-worked/ Sat, 08 Feb 2014 01:00:03 +0000 https://boyter.org/2014/02/worst-program-worked/ The worst program I ever worked on was something I was asked to maintain once. It consisted of two parts. The first was a web application writen in ASP. The second portion was essentially Microsoft Reporting Services implemented in 80,000 lines of VB.NET. The first thing I did was chuck it into VS2010 and run some code metrics on it. The results were, 10 or so Methods had 2000+ lines of code. https://boyter.org/2014/02/permission-clean-code/ Fri, 07 Feb 2014 01:00:06 +0000 https://boyter.org/2014/02/permission-clean-code/ This is something that took me 2 years or so to learn. One day I realised nobody was really looking at my timecards in depth so I started allocating extra time to things and using the extra time to fix the things I thought needed fixing. Once I started delivering on this I showed my manager who agreed that it was a good use of time. I was given free reign to fix anything I felt would add maximum value, provided the bug fixes continued to be delivered without any major compromise. https://boyter.org/2014/02/storing-tracking-managing-billions-tiny-files-file-system-nightmare/ Thu, 06 Feb 2014 01:00:59 +0000 https://boyter.org/2014/02/storing-tracking-managing-billions-tiny-files-file-system-nightmare/ Its a real pain when you want to inspect the files, delete or copy them. Try taking 300,000 files and copy them somewhere. Then copy 1 file which has the size of the 300,000 combined. The single file is MUCH faster (its also why we usually do a tar operation before copying stuff if its already compressed). Any database that’s not a toy will usually lay the 300,000 records out in a single file (depending on settings, sizes and filesystem limits). https://boyter.org/2014/02/counter-counter-argument-tdd/ Wed, 05 Feb 2014 01:00:25 +0000 https://boyter.org/2014/02/counter-counter-argument-tdd/ The following is taken from my response to a Hacker News comment. The comment follows (quoted) and my response below. “I will start doing TDD when, 1. It is faster than developing without it. 2. It doesn’t result in a ton of brittle tests that can’t survive an upgrade or massive change in the API that is already enough trouble to manage on the implementation-side- even though there may be no functional changes! https://boyter.org/china-trip-3/ Tue, 04 Feb 2014 22:12:04 +0000 https://boyter.org/china-trip-3/ 01/02/05 – 14/01/05 Well I figured I may as well add an end of the story, I believe that anyone who reads this deserves and end to the story and hence I am writing this for you all. Anyways what to say….. Here is my description of New Years Eve for you all. IT WAS LIKE THE APOCALYPSE!!!!!!!!!!!!!!!!!!!!!!!! Gunpowder smoke everywhere (making it hard to see), fires burning in the street, fireworks launching in the sky, enough small bangs that it sounds like its raining VERY hard on a tin roof. https://boyter.org/china-trip-2/ Tue, 04 Feb 2014 22:11:49 +0000 https://boyter.org/china-trip-2/ 3/11/04 – 5/11/04 Let me recap the conversation I had with Tiff on the 5th over the phone. “Hello?” “Tiff Sorry about waking you. Have a look out the window” “Ok if I have to… OH MY GOD!!! SNOW!!!! I NEED TO CALL MY MUM!” Yes that’s correct. On the 5th I awoke to a winter wonderland outside. Not much snow apparently but it was enough to get Tiff and I excited. https://boyter.org/china-trip/ Tue, 04 Feb 2014 22:11:19 +0000 https://boyter.org/china-trip/ Everything below is from when I was in University and went on exchange to Heilongjiang University in Harbin China. Please excuse the spelling and whatever is below as I did write it quite a while ago. This content is taken from http://silica.csu.edu.au/depot/dapeng/harbin/ as I want to preserve a copy just in case it is lost. I will be making a copy of the photos available at some point as there is a lot of content in there I would also like to preserve. https://boyter.org/2014/02/explain-regex-matches-ascii-characters/ Tue, 04 Feb 2014 00:07:02 +0000 https://boyter.org/2014/02/explain-regex-matches-ascii-characters/ Since I am pulling most of my content from other sites such as Mahalo and Quora I thought I would pull back some of my more interesting HN comments. Can anyone explain how this regex [- ~] matches ASCII characters? It’s pretty simple. Assuming you know regex… Im going to assume you don’t since you are asking. The bracket expression [ ] defines single characters to match, however you can have more then 1 character inside which all will match. https://boyter.org/2013/12/quora-answer-writing-search-engine/ Fri, 06 Dec 2013 01:31:30 +0000 https://boyter.org/2013/12/quora-answer-writing-search-engine/ The following I posted on Quora in response to the question “I am planning to make a small scale search engine on my local system, but I don’t know from where to start?”. It’s a reasonable answer so like my Mahalo one I thought I would make a copy for myself. I agree with Wolf Garbe and that you are better off in your case starting with existing technologies, have a look at http://yacy. https://boyter.org/2013/09/introducing-singlebugs-bug-tracker-single-developers/ Mon, 16 Sep 2013 01:50:08 +0000 https://boyter.org/2013/09/introducing-singlebugs-bug-tracker-single-developers/ Introducing the first beta release of SingleBugs, the bug tracker aimed at single/solo developers. Are you a solo developer? Do you find setting up Mantis/Fogbugs/Bugs.net/et.al too complex and a waste of your time? Do you want a single solution that easily syncs and backs up between your machines? Try SingleBugs. Guaranteed to save you time and money setting up a bug tracker. Guaranteed to be the fastest bug tracker you have ever used. https://boyter.org/2013/09/fizzbuzz-bug-tracker/ Fri, 06 Sep 2013 05:22:26 +0000 https://boyter.org/2013/09/fizzbuzz-bug-tracker/ This serves as the announcement of my new bug tracker product I am working on. My needs are pretty specific and none of the existing bug trackers I have tried have meet the following goals. Speed. Searching, adding projects/issues/comments should be instant. Any time waiting on the bug tracker is wasted time. Outlook style view of projects/issues/comments. This should allow me to get an overview of how I am tracking. Not require me to spin up a full web-server. https://boyter.org/2013/09/collection-letters-training/ Tue, 03 Sep 2013 05:59:42 +0000 https://boyter.org/2013/09/collection-letters-training/ I was looking for this on Google the other day and unable to find it. Essentially what I needed was a collection of images which are all the same size, but of different fonts so that I use them for training Neural Networks and test other OCR techniques. Since I couldn’t find any I thought I would upload my own collection. I used the below images when working on my thesis. https://boyter.org/2013/08/saving-resources-humans-php/ Tue, 27 Aug 2013 00:23:43 +0000 https://boyter.org/2013/08/saving-resources-humans-php/ One of the issues I have run into running searchcode.com is that a huge amount of time spent serving pages is serving them to bots (about 900,000 hits a day are from bots). I have been watching the load averages and they can spike to over 30.00 occasionally which is pretty bad for a 4 core/ht system. I don’t have any problems with bot’s really other then the fact that I cannot really control them. https://boyter.org/2013/08/c-vector-space-implementation/ Tue, 13 Aug 2013 05:59:09 +0000 https://boyter.org/2013/08/c-vector-space-implementation/ Since I am writing lots of Vector Space implementations in Go, Python etc… I thought I would add another one in C#. This one is a little more verbose then either the Python or Go implementations. The verbosity is mostly due to not using any of the nice C# LINQ functionality which would really reduce the size. In any case here it is in case you are looking for a simple implementation of this useful class. https://boyter.org/2013/08/golang-vector-space-implementation/ Mon, 12 Aug 2013 06:51:30 +0000 https://boyter.org/2013/08/golang-vector-space-implementation/ UPDATE – This is now actually available as a real Golang import with tests. Get it at https://github.com/boyter/golangvectorspace I have mentioned this before somewhere but one of the first things I usually attempt to implement in any programming language I want to play with is a vector space. Its my own personal FizzBuzz implementation. It usually covers everything I need to know in a language (imports, functions, string manipulation, math functions, iteration, maps etc…) so I consider it a good thing to get started with. https://boyter.org/2013/07/clean-repository-data-access-c/ Wed, 31 Jul 2013 23:52:14 +0000 https://boyter.org/2013/07/clean-repository-data-access-c/ Mostly as a self reference here is an extremely clean data access pattern possible using C# and Entity Framework. It saves you the effort of mocking the database context as the code you end up writing is so simple it is all compile time checked. Essentially you define a very simple class which provides a single method for getting data (although you may want a save data method too) and make sure you add an interface to make unit testing/mocking easier. https://boyter.org/2013/07/gigablast-aquired-code-posted/ Mon, 29 Jul 2013 05:58:30 +0000 https://boyter.org/2013/07/gigablast-aquired-code-posted/ Interestingly it seems that Matt Well’s search engine Gigablast has been acquired by Yippy.com [1]1 [2]2 [3]3 (demo here http://demo.yippy.com/). Gigablast has always been one of my favorite search engines simply because it is so interesting. Started by a single guy, with an interesting blog and being one of the last true new indexes of the web it was always worth a look. While its sad to see it go this way I am happy that Matt presumably has been able to cash out on his creation. https://boyter.org/2013/07/list-captcha-decoding-articles/ Thu, 18 Jul 2013 23:46:43 +0000 https://boyter.org/2013/07/list-captcha-decoding-articles/ This website ranks quite high in most search engines for the search term “captcha decoding” or some permutation of it. As such here are a collection of useful links if you are looking into doing such a thing. If any more come up I will be sure to update this post. http://www.boyter.org/decoding-captchas/ Shameless self promotion but this link is why this page ranks so highly. Its an article I wrote some time ago about how to go about decoding a simple CAPTCHA. https://boyter.org/about/ Thu, 11 Jul 2013 04:02:47 +0000 https://boyter.org/about/ I’m a software developer by trade who after years of leaving articles and blog posts all over the place has finally gotten around to consolidating them all into this single site. I am a professional software engineer who has been working in development with a focus on testing for over 15 years. In that time I have played an integral part in developing and implementing testing methodologies for all sorts of projects including, https://boyter.org/decoding-captchas/ Thu, 11 Jul 2013 03:00:02 +0000 https://boyter.org/decoding-captchas/ Looking for a practical guide to CAPTCHA decoding? All About CAPTCHA’s. This eBook will teach you how to identify weaknesses and exploit CAPTCHA’s from beginning to end. Buy now using Leanpub Most people don’t know this but my honors thesis was about using a computer program to read text out of web images. My theory was that if you could get a high level of successful extraction you could use it as another source of data which could be used to improve search engine results. https://boyter.org/2013/04/building-a-search-result-extract-generator-in-php/ Tue, 16 Apr 2013 23:23:56 +0000 https://boyter.org/2013/04/building-a-search-result-extract-generator-in-php/ During some contracting I was doing recently there was a requirement to implement some search logic using only PHP. There are no issues with that but it turns out I couldn’t find a decent extract generator handy as usually one would just plug into the search engines provided version to do this. Off the top of my head I could only think of one example I was aware of which lives in Sphider (for the record it lives in searchfuncs. https://boyter.org/2013/02/why-code-search-is-difficult/ Thu, 28 Feb 2013 23:26:36 +0000 https://boyter.org/2013/02/why-code-search-is-difficult/ I was chatting with a colleague the other day and he was asking me why code search is a difficult problem. After all its not quite as dynamic as the web so it should be easier to index. In truth its a mixed bag. Some things like like crawling are easy, others such as indexing are much harder then you would think. I thought I would write down why this is. https://boyter.org/2013/01/want-to-write-a-search-engine-have-some-links/ Wed, 30 Jan 2013 23:29:36 +0000 https://boyter.org/2013/01/want-to-write-a-search-engine-have-some-links/ A recent comment I left on Hacker News managed to get quite a lot of up-votes which surprised me since it was effectively just a collection of links about search engines. You can read the full thread at http://news.ycombinator.com/item?id=5129530 Anyway since it did do so well I thought I would flesh it out with some more information. Here are a collection of posts/blogs/discussions which go into the details of how to write a search engine. https://boyter.org/2013/01/code-for-a-search-engine-in-php-part-5/ Thu, 10 Jan 2013 05:34:18 +0000 https://boyter.org/2013/01/code-for-a-search-engine-in-php-part-5/ Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so. This is part 5 of a 5 part series. Part 1 – Part 2 – Part 3 – Part 4 – Part 5 – Downloads/Code So we need to convert the indexer to a method that wont consume as much memory. Looking at how it works now we can determine a few areas that could be improved before we implement out new method. https://boyter.org/2013/01/code-for-a-search-engine-in-php-part-4/ Thu, 10 Jan 2013 05:31:34 +0000 https://boyter.org/2013/01/code-for-a-search-engine-in-php-part-4/ Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so. This is part 4 of a 5 part series. Part 1 – Part 2 – Part 3 – Part 4 – Part 5 – Downloads/Code So previously we had 4 main issues identified. Im going to go with low hanging fruit and add a simple porn filter. https://boyter.org/2013/01/code-for-a-search-engine-in-php-part-3/ Thu, 10 Jan 2013 05:27:57 +0000 https://boyter.org/2013/01/code-for-a-search-engine-in-php-part-3/ Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so. This is part 3 of a 5 part series. Part 1 – Part 2 – Part 3 – Part 4 – Part 5 – Downloads/Code At this point I had a large chunk of the web sitting on my disk waiting to be indexed. https://boyter.org/2013/01/code-for-a-search-engine-in-php-part-2/ Thu, 10 Jan 2013 03:03:54 +0000 https://boyter.org/2013/01/code-for-a-search-engine-in-php-part-2/ Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so. This is part 2 of a 5 part series. Part 1 – Part 2 – Part 3 – Part 4 – Part 5 – Downloads/Code The first implementation has a few issues that are pretty apparent. Chiefly is performance. Secondly it stores the index in a single folder which is loosely related to the first. https://boyter.org/2013/01/code-for-a-search-engine-in-php-part-1/ Thu, 10 Jan 2013 02:55:54 +0000 https://boyter.org/2013/01/code-for-a-search-engine-in-php-part-1/ Are you really interested in learning how to Write a Search Engine from Scratch? Click this link and register your interest for my book about how to do so. This is part 1 of a 5 part series. Part 1 – Part 2 – Part 3 – Part 4 – Part 5 – Downloads/Code I imagine that if you have landed on this page you probably have an interest in search and search engines. https://boyter.org/2012/12/variety-of-updates/ Mon, 03 Dec 2012 23:40:37 +0000 https://boyter.org/2012/12/variety-of-updates/ A variety of updates have just rolled out. Firstly there is the new look and feel which I am actually proud of. The website looks far better, which wasn’t hard considering how awful it looked before. The other update is that documentation and code results are mixed together now. An example of this would be the search for mysql_query which displays the PHP reference which has mysql_query in it above the code results. https://boyter.org/2012/11/building-a-search-engine-the-most-important-feature-you-can-add/ Thu, 15 Nov 2012 23:21:48 +0000 https://boyter.org/2012/11/building-a-search-engine-the-most-important-feature-you-can-add/ TL/DR Part of the genius behind DuckDuckGo customer retention is the !bang feature. Every new search engine should implement it while trying to gain traction. Let me explain. Your current search engine is like a favorite couch. Its comfortable, familiar and you have been using it for years. Switching is not a comfortable experience. Im a search engine junkie. The moment I discover a new one I switch my defaults over to it and for a period of about a week use it day to day. https://boyter.org/2012/10/updates-in-the-works/ Fri, 12 Oct 2012 23:42:44 +0000 https://boyter.org/2012/10/updates-in-the-works/ Recently I have rolled out a few small performance and functional tweaks and changes. Nothing major to report here however I am working on the next version of searchcode which has an improved look and feel and blended search results. The idea being if you search for “mysql_query” you are probably interested in the documentation first and code examples second, or if you search for “mysql_query is a blocking function?” you are probably interested in web or stackoverflow results. https://boyter.org/2012/07/billions-of-lines-of-code/ Mon, 16 Jul 2012 23:43:44 +0000 https://boyter.org/2012/07/billions-of-lines-of-code/ Recently I fired off the spiders to pull in an additional 30,000 projects I was aware of and then added them to the index. The result being searchcode now has 3.8 billion lines of code indexed. The exact count being 3,863,756,553 lines. Of course the reason this is most exciting for me is that koders.com which is the default code search (now that Google Code search has been retired) has 3. https://boyter.org/2012/06/codesearch-api/ Tue, 26 Jun 2012 23:50:23 +0000 https://boyter.org/2012/06/codesearch-api/ Happy to announce that the CodeSearch API is now alive and kicking for those who wish to use it. The details are included below. Example calls, JSON http://searchco.de/api/codesearch_I/?q=%23define http://searchco.de/api/codesearch_I/?q=IO::pipe http://searchco.de/api/codesearch_I/?q=goto%20lang:perl http://searchco.de/api/codesearch_I/?q=test%20lang:perl&p=1 JSONP http://searchco.de/api/jsonp_codesearch_I/?q=%23define&callback=CALLBACK http://searchco.de/api/jsonp_codesearch_I/?q=IO::pipe&callback=CALLBACK http://searchco.de/api/jsonp_codesearch_I/?q=goto%20lang:perl&callback=CALLBACK http://searchco.de/api/jsonp_codesearch_I/?q=test%20lang:perl&p=1&callback=CALLBACK Parameters are, q = query p = page The return results are pretty self explanatory, except that they include a modeltype which indicates what sort of match it was, the types being exactmatch, closematch, regexexact, regexclose https://boyter.org/2012/05/running-the-numbers/ Tue, 29 May 2012 23:53:54 +0000 https://boyter.org/2012/05/running-the-numbers/ So I have rolled out quite a few performance improvements. searchcode is MUCH MUCH faster then it was before. I also added some various improvements across the board in terms of relevance. This included indexing characters like !@#$%^&*()-= etc… So now things like the perl regex match =~ is now a valid search term. Of course you can combine terms and normal characters to get really complex search terms such as $localdate =~ /([0-9]+):([0-9]+):([0-9]+)/; Pretty awesome stuff I think. https://boyter.org/2012/05/who-knows-regex/ Sun, 13 May 2012 05:25:10 +0000 https://boyter.org/2012/05/who-knows-regex/ Apparently not many. I have been monitoring how the search has been used since I rolled out code search and noticed that most people are just typing in search terms and not regex search terms. Of course this means some results are not what people are expecting. I have thus changed the way searches work. It now does an exact match of whatever it is you are looking for UNLESS you wrap your search term in / in which case it will default to a regex search. https://boyter.org/2012/05/expanded-syntax-lang-keyword-now-supported/ Tue, 01 May 2012 05:15:11 +0000 https://boyter.org/2012/05/expanded-syntax-lang-keyword-now-supported/ Trawling through the logs of search queries I noticed that some people are using the Google Code Search lang syntax. An example that I spotted was the following “throw.* runtime_error lang:c++” Note the lang:c++ portion. Of couse this ended up spitting back no useful results because the lang:c++ was treated as part of the search. Well no longer is this the case. searchcode now supports the lang keyword in addition to the existing ext one (useful for extensions). https://boyter.org/2012/04/growing-index/ Fri, 13 Apr 2012 05:13:12 +0000 https://boyter.org/2012/04/growing-index/ Happy to announce that the searchco.de code index is growing nicely. Even as you read this blog (assuming anyone does read it) thousands of new files are being added to the index. I suspect in time the code index will be one of the larger ones on the web. At last count it was well over 2 billion lines of code but I suspect this has grown quite a lot. https://boyter.org/2012/04/performance/ Thu, 12 Apr 2012 05:09:00 +0000 https://boyter.org/2012/04/performance/ Just a quick blog post to point out that some speed improvements have been implemented. I have a few more tricks to roll out soon but on the whole things are much more snappy now. I have also been modifying my indexers and can now update the index ad-hoc rather then in large batches. I have some more stuff coming soon including search restricted to languages EG Java / C++ and an updated code view. https://boyter.org/2012/02/improving-the-index/ Wed, 29 Feb 2012 05:07:36 +0000 https://boyter.org/2012/02/improving-the-index/ The last couple of weeks I have been spending improving the index based on feedback about duplicate results. When indexing source code you will find a lot of duplicate libraries and copied code all over the place. Usually when searching for something like jQuery you don’t want to see thousands of results of the same file. However when searching for method calls you do want to see results which call the method in a similar way as you may be looking for examples of how to use it. https://boyter.org/2012/02/growing-pains/ Mon, 06 Feb 2012 05:04:27 +0000 https://boyter.org/2012/02/growing-pains/ When I started searchco.de the amount of hardware required was fairly low. I did a lot of processing on other machines and pushed the results through allowing the machine that serves the site to just focus on the job of delivering results. The result was everything was running on a nice VPS provided by Atum.com 2.9 Ghz CPU 512 MB ram and about 30 GB of hard disk space. This was more then enough to support the index I had and do everything required. https://boyter.org/2012/01/updates-coming-soon/ Fri, 13 Jan 2012 05:01:49 +0000 https://boyter.org/2012/01/updates-coming-soon/ A few updates should be rolling out soon to searchco.de with the first being a big jump on the amount of open source/free software code being indexed. At last check there was about 1.1 billion lines of code indexed. This should more then double in the next release to well over 2 billion. When I have an exact count I will publish it here. Secondly some massive speed increases are on the way. https://boyter.org/2012/01/mutation-tester-for-all-languages/ Fri, 06 Jan 2012 04:59:42 +0000 https://boyter.org/2012/01/mutation-tester-for-all-languages/ As a fan of unit tests for bug checking and development (where applicable) I always have a doubt that even though my tests pass they might not be written correctly. This is especially true where you write the tests after development rather then following TDD. Regardless you can still stress your code by doing some mutation testing. It’s essentially a way of testing your tests. Quite a few mutation testing frameworks are out there such as Heckle, Insure++, Nester etc. https://boyter.org/2012/01/errors-in-search/ Wed, 04 Jan 2012 04:55:07 +0000 https://boyter.org/2012/01/errors-in-search/ EDIT – This has now been resolved. All the below searches should work correctly, with the exception of XCompositeGetOverlayWindow. I am adding that to the index to be refreshed sometime in the next month or so. Well thanks to some sample searches being thrown against the codesearch index I can finally start tuning issues that have cropped up. The main issue I have currently is searches returning no results where you would expect some. https://boyter.org/2011/12/searchcode-now-supports-regex-code-search/ Sat, 17 Dec 2011 04:52:21 +0000 https://boyter.org/2011/12/searchcode-now-supports-regex-code-search/ searchcode updates now supports regex code search Similar to the soon to be defunt Google Code Search you can now search over 1 billion lines of code from github, bitbucket, sourceforge, google code and codeplex on searchcode. All you need do it search using a regex such as /[cb]at/ or /a{1,5}/ etc… You can filter by file extension as well using ext:[FILETYPE] EG, [/print_r/ ext:php][3] The only catch is due to constraints you can only do prefix regex not infix. https://boyter.org/2011/11/wildcards/ Mon, 28 Nov 2011 04:49:22 +0000 https://boyter.org/2011/11/wildcards/ Just a quick update to let you know that wildcard searches have been updated to support prefix and postfix. Essentially this means the following searches now all work, *print *clo* clo* In addition code characters such as { } [ ] < > are now being indexed. Finally searchco.de has been on a bit of a diet and should be a lot faster to load and search. https://boyter.org/2011/10/hello-world-collection/ Tue, 18 Oct 2011 04:48:09 +0000 https://boyter.org/2011/10/hello-world-collection/ I was looking through the logs of what people are searching for and noticed there is a large amount of “Hello World” searches such as “hello world java” which actually came up with no results. I couldn’t have that continue so I quickly integrated the Hello World Collection You can now search for hello world in 400+ languages (although curiously no in Clojure). Just type hello world and the language you are searching for EG, https://boyter.org/2011/10/underscorejs-joins-the-party/ Mon, 17 Oct 2011 04:45:28 +0000 https://boyter.org/2011/10/underscorejs-joins-the-party/ Had a spare moment and added underscore.js documentation in. underscore.js template _ pluck Still working on C++ Ruby and C#. I will be adding backbone.js soon however as I am using that on my own projects. https://boyter.org/2011/10/google-killing-off-code-search/ Sat, 15 Oct 2011 04:43:32 +0000 https://boyter.org/2011/10/google-killing-off-code-search/ Google has just annouced that they are killing off two new products, Buzz and Google Code search (read here). Buzz is a no brainer (although I am certain someone out there is using it) but code search is a little sad. Its also both annoying and an opportunity for http://searchco.de/ Its annoying because I have branch in my source control which integrates with Google code search. I was about to push it out too. https://boyter.org/2011/10/coverage/ Mon, 10 Oct 2011 04:41:06 +0000 https://boyter.org/2011/10/coverage/ It was a long weekend here in Australia and as such I decided to take it with my girlfriend and spend time totally off the grid. As I came back from my time off I had a look at my general stats and was very surprised to see some massive spikes. Looked into it a bit more and it seems searchco.de has gotten some coverage from a few sites. See the below, https://boyter.org/2011/09/slow/ Tue, 06 Sep 2011 04:38:11 +0000 https://boyter.org/2011/09/slow/ Well it seems searchco.de has been very slow to respond over the last few days. Turns out I had some rogue processes running for another domain chunews.com which were eating a lot of the CPU. I just killed them and switched off the processes for maybe good as that website really needs an overhaul anyway. https://boyter.org/2011/08/hard-numbers/ Thu, 04 Aug 2011 04:29:46 +0000 https://boyter.org/2011/08/hard-numbers/ Thought I would quickly post some hard numbers showing the number of searches since searchco.de launched. The results surprised me somewhat actually. 2011-7 13636 2011-6 8154 2011-5 6183 2011-4 8532 2011-3 10886 2011-2 6476 2011-1 948 I could dig into the results more closely since a lot of those could be the ajax firing off events like “a” and the like but I do like the upwardish trend. https://boyter.org/2011/06/vector-space-search-model-explained/ Tue, 28 Jun 2011 23:19:48 +0000 https://boyter.org/2011/06/vector-space-search-model-explained/ A mate of mine was looking at a previous article I wrote about Decoding CAPTCHA’s where I pointed people to the following article (PDF) http://la2600.org/talks/files/20040102/Vector_Space_Search_Engine_Theory.pdf He was having some difficulty understanding it so I thought I would write up a very simple explanation of what’s actually happening in the vector space. The vector space isn’t actually that complicated, but getting your head around how it works takes a few steps. Let’s imagine for the moment that we are going to search over a collection of documents which only contain one word “boffin”. https://boyter.org/2011/06/grep-match-a-tab/ Tue, 07 Jun 2011 23:19:24 +0000 https://boyter.org/2011/06/grep-match-a-tab/ Ever wanted to match a tab while using grep for some reason? The trick (under bash anyway) is to Ctrl+V and then press the tab key so you get whatever you are looking for. $ cat file_to_grep.txt | grep "^log " I was trying to match a file for the exact match of log and then a tab. Without the tab I ended up getting back a bunch of junk results like “logger” “logging” “login” etc… https://boyter.org/2011/06/clojure-13-now-avaliable/ Mon, 06 Jun 2011 23:19:04 +0000 https://boyter.org/2011/06/clojure-13-now-avaliable/ Added Clojure 1.3 documentation today. The Clojure language itself is pretty tight, but its nice to have something to browse through when looking for some elusive method. You can view it by searching for clojure One thing I have discovered while doing this is that if I have a slight amount of familiarity with the language it really makes parsing it easier. This stands to reason as I can pick up errors more easily, but somewhat surprising. https://boyter.org/2011/06/link-love/ Wed, 01 Jun 2011 23:18:41 +0000 https://boyter.org/2011/06/link-love/ With the fall from grace of the TWiT podcast (less Dvorak and no Calacanis makes for boring shows) I went looking for new podcast’s to keep me entertained over the last couple of months. Here are a few that I highly recommend. Tech Podcast TechZing Live The boys from tech-zing are full of energy, always come up with new stuff and usually manage to do one thing technical each show that makes me want to scream with frustration. https://boyter.org/2011/05/nginx-httpcoremodule-documentation/ Fri, 20 May 2011 23:18:17 +0000 https://boyter.org/2011/05/nginx-httpcoremodule-documentation/ Added the Nginx HttpCoreModule Documentation to the index. Hopefully helps someone out there! You can view it here Nginx HttpCoreModule Documentation https://boyter.org/2011/04/jquery-hello-world-plugin/ Fri, 15 Apr 2011 23:17:36 +0000 https://boyter.org/2011/04/jquery-hello-world-plugin/ I was doing a simple job test the other day and one of the questions involved creating a simple jQuery plugin. Having never created one myself I had to look into how to do it. I couldn’t find a dead simple hello world plugin example so I thought I would create a simple example here for people to look at. (function( $ ){ $.fn.HelloWorld = function() { $(this).html('Hello World!'); }; })( jQuery ); The above essentially just attaches a new function called HelloWorld to the basic jQuery object. https://boyter.org/2011/04/jquery-documention-now-avaliable/ Wed, 13 Apr 2011 23:16:56 +0000 https://boyter.org/2011/04/jquery-documention-now-avaliable/ It has been a long time coming, but jQuery is now in the index and can be searched on. You can by searching for jquery EDIT And a list of all functions to go with it, List of all jQuery Functions https://boyter.org/2011/03/store-enumerations-database/ Wed, 30 Mar 2011 23:16:26 +0000 https://boyter.org/2011/03/store-enumerations-database/ One of the things I come across in databases now and then is a collection of single tables with a name like “MessageType”. You have a look in them and it turns out to have 6 or so rows with no foreign key relationships. Every single time it turns out to be someone had the idea to store an Enumeration (Enum) type in the database. Not a bad idea as it turns out since you can add sort options, soft deletes and the like, but the implementation of a single table for each one is flawed. https://boyter.org/2011/03/lesson-learnt-save-assets/ Mon, 28 Mar 2011 23:15:49 +0000 https://boyter.org/2011/03/lesson-learnt-save-assets/ I really should learn from my past mistakes, and usually I do, but this one comes to bite me again and again. I never save assets. This results in me having to find fonts, recreate images and the like over and over. If you do a hard refresh of this page (CTRL+F5) you will notice the logo looks a little different. This is because I couldn’t find the font I used before and had to find something similar. https://boyter.org/2011/03/ios-documention-now-avaliable/ Sun, 20 Mar 2011 23:15:12 +0000 https://boyter.org/2011/03/ios-documention-now-avaliable/ Added iOS documentation to join the OSX documentation. You can by searching for appleios https://boyter.org/2011/03/lorem-ipsum-word-generation/ Thu, 17 Mar 2011 23:14:28 +0000 https://boyter.org/2011/03/lorem-ipsum-word-generation/ Something I tend to need from time to time is some generated lorem ipsum text as a placeholder. I would normally search for lorem ipsum, click through to lipsum.com click paragraph etc… I thought this was too much work, so I quickly added a lipsum generator. To use it just type the following, lorem:100 Where you can replace 100 with any number of words up to 1000. Works for the following search texts as well, https://boyter.org/2011/03/list-of-most-commonly-used-php-functions/ Tue, 15 Mar 2011 23:13:43 +0000 https://boyter.org/2011/03/list-of-most-commonly-used-php-functions/ One of the things about ranking in Search is that you need to consider all sorts of methods of working out what is relevent. Google broke new ground (although the idea had already existed) with its PageRank algorithm which supplied better search results then all the other search engines. For what I am doing however I need to consider what programmers are looking for. One thing that I considered some time ago was working out which are the most common functions in a language and adding this as an additional signal to ranking. https://boyter.org/2011/03/list-of-mysql-sql-server-2008-and-apache-directives/ Mon, 14 Mar 2011 23:13:47 +0000 https://boyter.org/2011/03/list-of-mysql-sql-server-2008-and-apache-directives/ Added some more lists now that the feature is working correctly. List of MySQL Functions List of SQL Server 2008 Functions List of Apache 2 Directives https://boyter.org/2011/03/list-of-all-gnu-linux-windows-commands/ Mon, 14 Mar 2011 23:11:51 +0000 https://boyter.org/2011/03/list-of-all-gnu-linux-windows-commands/ Today I can announce a small but possibly useful feature which is the addition of lists. They are here as way of making it easier to find what you are looking for if the main search lets you down and you want to try a browser inline search. They can also be useful if you just want to scroll through everything in a particular subject. I have the following lists already created, https://boyter.org/2011/03/batf-big-arse-text-file/ Mon, 07 Mar 2011 23:10:23 +0000 https://boyter.org/2011/03/batf-big-arse-text-file/ Ever needed the ability to track bugs and features without using a full featured bug/feature tracker? What about storing all your random notes such as server details, blog ideas, books to read, urls etc, without using a full featured CMS or the like. Want to have everything searchable and in the most platform independent format possible? Enter the BATF. I have always been a fan of the big arse text file (BATF) for keeping track of the above. https://boyter.org/2011/02/wget-xargs/ Mon, 28 Feb 2011 05:46:55 +0000 https://boyter.org/2011/02/wget-xargs/ The joy of the linux/unix command line is how versatile the commands are. I recently had 50,000 URL’s I needed to download in a text file. I was thinking about writing a crawler in Python to do it but ended up just doing the following, cat urllist | xargs -P16 wget -i A 16 thread (process really) webcrawler in a single command. Joy. https://boyter.org/2011/02/mysql-export-csv/ Mon, 14 Feb 2011 23:33:51 +0000 https://boyter.org/2011/02/mysql-export-csv/ Ever needed to export data from MySQL into a CSV file? Its actually fairly simple, SELECT * INTO OUTFILE '/tmp/name.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\' LINES TERMINATED BY '\n' FROM [tablename] Certainly easier then writing a quick Python/Perl/PHP script to do the job. https://boyter.org/2011/01/mysql-command-line-import-utf-8/ Thu, 27 Jan 2011 21:24:36 +0000 https://boyter.org/2011/01/mysql-command-line-import-utf-8/ Ever wanted to command line import some data into MySQL and keep the encoding type? Turns out its not that difficult. Just a simple command line option. That said I have to look it up all the time. mysql -u USERNAME -pPASSWORD --default_character_set utf8 DATABASE > file.sql That will import things across with the correct encoding type. I think personally that the face that MySQL fails to throw an error or even raise a warning when it encounters these sort of issues is wrong but where you have no choice the above fixes problems. https://boyter.org/2010/12/mysql-popularity-ranking-algorithm/ Fri, 24 Dec 2010 04:11:03 +0000 https://boyter.org/2010/12/mysql-popularity-ranking-algorithm/ Calculating the popularity of a page or article is something that usually comes up as a list of requirements for any social website. Essentially you want to display the post popular items/articles in some form of list but have them weighted by how old they are. Thankfully its pretty easy to do MySQL. ((popularity-1)/power(((unix_timestamp(NOW())-unix_timestamp(datetime))/60)/60,1.8))``` The above produces a number which you can then sort on. It is based on the Hacker News algorithm and works well for items which change hourly. https://boyter.org/2010/11/web-bigger/ Wed, 03 Nov 2010 22:15:47 +0000 https://boyter.org/2010/11/web-bigger/ About two years ago Google announced (probably to take some of the wind out of Cuil’s sails) that they had found over 1 trillion unique URL’s on the web. Keep in mind that not all of those are unique as some pages could have multiple URL’s but its still a mind bottlingly large number. I never really realised the scale of the web though till I started crawling it myself. Not armed with billions, or even millions of dollars I am doing this on the cheap but you can still suck down a large amount of stuff. https://boyter.org/2010/10/minimum-viable-product-mvp/ Sun, 17 Oct 2010 08:36:27 +0000 https://boyter.org/2010/10/minimum-viable-product-mvp/ As much as I like things to be perfect I soon discovered from my startup camp experience that its far more important to have a minimum viable product out there and generating interest. After all how can you determine how worthwhile your time investment is without throwing out your ideas and testing the reaction. That’s always something scary for any developer, but probably is the correct approach. And so I am releasing a quick weekend project I created when I became interested in Google’s Instant search. https://boyter.org/2010/10/mysql-error-error1356-view-references-invalid/ Thu, 14 Oct 2010 02:50:45 +0000 https://boyter.org/2010/10/mysql-error-error1356-view-references-invalid/ I managed to get the following error the other day while helping migrate and upgrade some MySQL databases for a client. mysqldump: Got error: 1356: View 'database.table' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them when using LOCK TABLES The above appeared anytime I tried to dump the database to a SQL file using mysqldump. Its actually one of the better errors I have seen come out of MySQL (which has pretty terrible error reporting most of the time) and tells you exactly what is going wrong. https://boyter.org/2010/10/startupcamp-opinions/ Wed, 13 Oct 2010 02:43:27 +0000 https://boyter.org/2010/10/startupcamp-opinions/ So the other weekend I attended a Sydney Startup Camp. While I am not going to say that the entire experience was positive im not going to call out the reasons in too much depth. What I have noticed is that its very ‘clicky’. It seems that despite the organizers attempt to break up up people who know each other it seems to be impossible. A lot of the experience seems to focus on the non technical side of things which is good for people like me who are mostly technical it also means that you end up having a marathon sprint to finish features and then spend a lot of time trying to keep awake while going though things that I would describe as about as interesting as watching paint dry. https://boyter.org/2010/09/python-snippet/ Wed, 29 Sep 2010 06:53:24 +0000 https://boyter.org/2010/09/python-snippet/ The below is a quick Python snippet which I use on a day to day basis for weeks, then promptly forget. Essentially its reading from standard input and then doing something with it. Very useful when you are trying to process data on the command line and have forgotten how to use awk/sed properly and grep has run out of steam. import sys import re for line in sys.stdin: values = line. https://boyter.org/2010/09/cuil-33-million-funding/ Mon, 20 Sep 2010 04:17:57 +0000 https://boyter.org/2010/09/cuil-33-million-funding/ Well it appears that Cuil the troubled search engine that didn’t is dead. While I am not that surprised by this considering its lackluster results, I do feel that this is bad for the web in general. With the Yahoo/Bing deal we now have very few independent indexes that power search on the web. The big players are down to the following it would seem, Google Bing Ask Gigablast Blekko What did suprise me about this though was the question people were asking about how the founders of Cuil ever managed to secure $33 million in funding. https://boyter.org/2010/09/php-entity-generator/ Wed, 15 Sep 2010 00:22:11 +0000 https://boyter.org/2010/09/php-entity-generator/ A while ago I was using the Django Framework and was a big fan of the parts of it which save time. One part which I both loved and hated was the ORM. The bit I loved was for creating new database entities, loading them and modifying them. The ability to just load up an object and modify it and then call its save method saved me a lot of time. https://boyter.org/2010/09/googles-colossus-explained/ Sun, 12 Sep 2010 23:07:20 +0000 https://boyter.org/2010/09/googles-colossus-explained/ So Google has called their new indexing system Caffeine, which is powered by Google’s BigTable, or as they call it internally Colossus. I guess now all we need is Microsoft to announce their Bing back-end is called “Guardian” and the world is over. Actually looking at all of the information they have shows that while everyone was chasing MapReduce that Google was looking at implementing a distributed database where each update/trigger implements an update to the index. https://boyter.org/2010/09/failure/ Mon, 06 Sep 2010 23:20:36 +0000 https://boyter.org/2010/09/failure/ So I guess now is about the time that I write about my first failure. Although I realised that the project was a failure quite a while ago I never wrote anything about it admitting so. I guess this can be considered my cleansing moment. So about a year ago when everyone was jumping on the Twitter bandwagon I remember reading about a simple app called MyTwitterButler that a .NET developer coded up in a few hours and was selling for $10. https://boyter.org/2010/09/small-steps-2-%E2%80%93-teaching-neural-network-learn-letter-b-z/ Sun, 05 Sep 2010 22:50:02 +0000 https://boyter.org/2010/09/small-steps-2-%E2%80%93-teaching-neural-network-learn-letter-b-z/ So in the previous article we managed to get our neural network to learn the difference between A and B. I mentioned at the end I was going to next test and teach it on various versions of A and B to see how effective it is, but rather then that I figured teaching a network to learn A from every other letter would be more interesting. Get the source to everything below in Step2 https://boyter.org/2010/08/small-steps-teaching-neural-network-learn-letter/ Tue, 31 Aug 2010 00:48:14 +0000 https://boyter.org/2010/08/small-steps-teaching-neural-network-learn-letter/ I’m going to make the assumption that if you are reading this you already know what a NN is, and you are trying to do some sort of image recognition. I’m also going to assume you are somewhat familiar with programming preferably in Python since that’s what all the examples will be using. Get the source to everything below in Step1.zip To get started we are going to need the following, https://boyter.org/2010/08/principles/ Wed, 25 Aug 2010 21:46:16 +0000 https://boyter.org/2010/08/principles/ Recently I was having an issue with some code I was working on for my pet project (A website search solution). Essentially my problem was that Smarty PHP wouldn’t loop through an array I had passed in. After much swearing and complaining I decided to take a step back and run through all of the newbie mistakes. In other words I looked at the problem from first principles. Turns out the issue was a missing $ before the variable I was trying to loop over. https://boyter.org/2010/08/build-vector-space-search-engine-python/ Mon, 23 Aug 2010 22:44:20 +0000 https://boyter.org/2010/08/build-vector-space-search-engine-python/ Ever wanted to code a search engine from scratch? Well actually its a pretty simple thing to do. Here is an example indexer I coded up in less then an hour using Python. The first thing we need to do is have a way to take our documents we want to search on and turn them into an concordance. A concordance for those not in the know is a count of every word that occurs in a document. https://boyter.org/2010/08/captchas-never-use-numbers-0-1-5-7/ Wed, 18 Aug 2010 23:26:21 +0000 https://boyter.org/2010/08/captchas-never-use-numbers-0-1-5-7/ Interestingly this sort of question pops up a lot in my referring search term stats. Why CAPTCHA’s never use the numbers 0 1 5 7 Its a relativity simple question with a reasonably simple answer. Its because each of the above numbers are easy to confuse with a letter. See the below, Are you able to tell the difference? For some yes, others, certainly not. For those wondering the first character is the number and the rest are letters. https://boyter.org/2010/08/setting-git-follow-subversion-workflow/ Wed, 18 Aug 2010 00:44:49 +0000 https://boyter.org/2010/08/setting-git-follow-subversion-workflow/ Moving from Subversion SVN to GIT can be a little confusing at first. I think the biggest thing I noticed was that GIT doesn’t have a specific work-flow; you have to pick your own. Personally I wanted to stick to my Subversion like work-flow with a central server which all my machines would pull and push too. Since it took a while to set up I thought I would throw up a blog post on how to do it. https://boyter.org/2010/08/why-you-shouldnt-roll-your-own-captcha/ Mon, 16 Aug 2010 00:04:02 +0000 https://boyter.org/2010/08/why-you-shouldnt-roll-your-own-captcha/ At a TechEd I attended a few years ago I was watching a presentation about Security presented by Rocky Heckman (read his blog its quite good). In it he was talking about security algorithms. The part that really stuck with me went like this, “Don’t write your own Crypto algorithms unless you have a Doctorate in Cryptography.” Interestingly someone there did have said qualification, and Rocky had to make an exception for that single person. https://boyter.org/2010/08/mysql-backups-done-easily/ Sun, 15 Aug 2010 02:33:25 +0000 https://boyter.org/2010/08/mysql-backups-done-easily/ One thing that comes up a lot on sites like Stackoverflow and the like is how to backup MySQL databases.The first answer is usually use mysqldump. This is all fine and good, till you start to want to dump multiple databases. You can do this all in one like using the –all-databases option however this makes restoring a single database an issue, since you have to parse out the parts you want which can be a pain. https://boyter.org/2010/08/richard-stallman-to-visit-australia/ Wed, 11 Aug 2010 22:56:37 +0000 https://boyter.org/2010/08/richard-stallman-to-visit-australia/ Im not usually one to promote events and the like unless I feel there is a genuine benefit to be had by attending but this is one stands out. Richard M Stallman, the guru of Free Software is coming Down Under to hold a talk. You can read about him here, Open Source Celebrity to visit Australia You can register for the event at the ACS Website I am certainly planning on attending. https://boyter.org/2010/08/at-scale-you-will-hit-every-performance-issue/ Wed, 11 Aug 2010 11:40:31 +0000 https://boyter.org/2010/08/at-scale-you-will-hit-every-performance-issue/ I used to think I knew a bit about performance, scalability and how to keep things trucking when you hit large amounts of data. Truth is I know diddly squat on the subject since the most I have ever done is read about how its done. To understand how I came about realising this you need some background. Essentially what I have been working on and hope to launch soon is a highly vertical search engine that websites can employ on their site and get highly relevant search results. https://boyter.org/2010/08/examples-of-bad-website-search/ Sat, 07 Aug 2010 04:38:37 +0000 https://boyter.org/2010/08/examples-of-bad-website-search/ I thought I would post some examples of websites with bad website search. Rather then pick on some small blogs lets go after the big boys. Starting with eBay. “There was a screenshot of ebay here at one point but now it is lost for all time..” The link to the left is an example of a simple search I performed on eBay for an iPad. The search comes back pretty quickly which is good but has a few issues I will go through now. https://boyter.org/2010/08/test-driven-development/ Fri, 06 Aug 2010 11:07:13 +0000 https://boyter.org/2010/08/test-driven-development/ I was reading a blog post today linked from Dzone which alluded to test driven development being a waste of time I am one of those people who has caught the test driven development (TDD) bug and I am a big fan of TDD and unit testing in general. In fact I am the one who posted the first comment on the linked blog and prompted the authors change to it. https://boyter.org/2010/08/why-writing-a-web-crawler-isnt-easy/ Thu, 05 Aug 2010 10:38:32 +0000 https://boyter.org/2010/08/why-writing-a-web-crawler-isnt-easy/ The below is just a few things to keep in mind if you are writing a crawler or considering writing one. The first is to assume that all of the links you have are broken to begin with. When I write this I dont mean assume that the link goes nowhere, but that the URL itself is actually wrong. Even when you seed your own list (like I did) you can still get some bad URL’s in there. https://boyter.org/2010/08/mysql-exporting-all-databases/ Wed, 04 Aug 2010 10:51:13 +0000 https://boyter.org/2010/08/mysql-exporting-all-databases/ One of the things that I always have to look up (when doing it manually that is) is how to export specific databases or all of them from MySQL using mysqldump. To avoid having to Google around every time I need the commands I thought I would preserve it here. Export a database from MySQL in one line NB password must follow -p without a space mysqldump -u user -pPASSWORD mydatabase > mydatabase. https://boyter.org/2010/08/php-bug-trackers/ Tue, 03 Aug 2010 08:55:31 +0000 https://boyter.org/2010/08/php-bug-trackers/ So in keeping with the lean startup style, I needed to work out which version control system and which bug tracker I would use. Internally at work I use subversion and JIRA. Both do the job quite well but I have found some shortcomings with subversion and JIRA is needlessly complicated for a single developer at the moment. So I looked for alturnatives. Version control was easier of the two to pick. https://boyter.org/2010/08/hello-world/ Mon, 02 Aug 2010 08:56:25 +0000 https://boyter.org/2010/08/hello-world/ Well I finally got around to doing what I always said I would. That being launch and online business. Inspired by all of the stories and discussions at news.ycombinator.com and in particular this single post http://news.ycombinator.com/item?id=1560422 I decided to finish off those projects that I have been working on for a while and launch. Naturally following the experience of the greats who have done it themselves, such as Patrick with Bingo Card Creator and Peldi of Balsamiq I thought I would throw up a blog, pick a theme quickly and get started. https://boyter.org/2009/08/impressed/ Wed, 19 Aug 2009 00:04:10 +0000 https://boyter.org/2009/08/impressed/ Was playing around with Python for a second there. I learnt about one of the new “multiprocessing” features. Its pretty standard stuff, but thankfully does what I actually wanted Python to always do. Allow me to multiprocess any Map function. Below is the sample code (2.6 and above only sorry). from multiprocessing import Pool import time def f(x): return x*x if __name__ == '__main__': p = Pool(processes=2) r = range(1,10000000) t = time. https://boyter.org/2009/06/interface-inheritance/ Thu, 04 Jun 2009 00:06:03 +0000 https://boyter.org/2009/06/interface-inheritance/ Just found this while testing http://www.bing.com/ and thought I should preserve it here to be dammed sure I can find if when I want it. If the relationship is clearly “is-a”, I use inheritance. If it is more like “can-be”, I use interfaces. Eg, TextBox “is-a” Control, ArrayList “can-be” enumerated (so it implements IEnumerable). https://boyter.org/2009/03/linq/ Thu, 26 Mar 2009 00:07:14 +0000 https://boyter.org/2009/03/linq/ Thought while my code was compiling and being tested I would throw up some thoughts about it. For those who don’t know you should probably go and read something else, since what follows is going to be pretty technical. So LINQ to SQL. At first I was totally opposed to it. The idea of taking a language that most developers know (SQL) and morphing it into something that .NET can use and slightly resembles SQL didn’t really appeal to me. https://boyter.org/2009/01/athlon-64-pc-512-ram-hold-home-server/ Wed, 28 Jan 2009 01:21:59 +0000 https://boyter.org/2009/01/athlon-64-pc-512-ram-hold-home-server/ This post was taken from a Mahalo Answers Question I answered. I thought rather then risk Mahalo going away (since Jason Calacanis has said hes working on something else) I thought I would copy it here. Question How will a Athlon 64 PC with 512 of ram hold up as a home server. Whats the best setup… read details. I want to setup a drag and drop file server as well as a voip server like teamspeak. https://boyter.org/2008/09/data-mining/ Mon, 22 Sep 2008 00:10:25 +0000 https://boyter.org/2008/09/data-mining/ So a quick update. I spent 2 hours or so yesterday working on the netflix data mining. I basically just tidied it up and started testing. I was almost finished and was about to post a blog with results, but decided I wasn’t happy with the results. It turned out there was a bug but that my code which worked out how related users are wasn’t as effective as I thought it would be. https://boyter.org/2008/02/sql-fun/ Tue, 26 Feb 2008 00:15:12 +0000 https://boyter.org/2008/02/sql-fun/ Well today while pondering things at lunch I was looking into the performance of SQL queries one stood out. That being the performance of getting random rows out of a database. Most people who have a basic understanding of MySQL (the database I use at home because its easy to set up, though I am looking at Posgresql) would say oh, to get random rows you do this, SELECT * FROM \`table\` ORDER BY rand() LIMIT 0,5 Which isn’t very efficient.