Bitsnoop — The Most Advanced Torrent Search in the World
Bitsnoop is the most advanced torrent search engine in the world. And one of the biggest — millions of torrents and petabytes of files indexed. BTek3 is a codename for our current systems' architecture.
Better, Neater and More Usable
First of all, thank you! You've been sending us tons of ideas on how to make our site better — and not a single one is lost. It's all here, in new Bitsnoop.
We're recreated user interface from scratch. Completely. Cleaner and handier — with the same minimalist looks loved by our users.
Magnet links exclusively now. No need to fiddle about with .torrent files any more! Check out Torrent Howto if you're not familiar with Magnet links.
Major improvements in query language. Better search algorithms, more options.
For users who do not want to see adult content (enabled by default). It's OK to leave it on — robots will ask you if you're searching for adult stuff.
List of torrents you've viewed recently. Stored in your browser's "cookies", you can delete it at any time.
Improved TV Finder
Our TV Finder was awesome already, now it got better. More shows and easier to use!
Automatic torrent validation and FakeSkan
It's like we've coded P2P power-user into our robots. Every torrent we index is validated and analyzed by robots who "vote" for torrent. This affects torrent's position in search results and can even flag it as "verified" or "fake".
New "Top 100" Ratings
Completely redesigned — more filtering, separate daily / weekly / monthly ratings.
We've introduced new tracker discovery / analysis algorithms. Faster and with much better results.
Bitsnoop is going Global
Full Unicode support. Now you are able to find torrents in any language, from Russian to Chinese. We're adding lots of international torrent sources for robots to crawl.
New site is like 40% faster. We did an extensive re-engineering of how stuff work at Bitsnoop — index compression, better algorithms, in-memory caching (for example, each of your searches passes through about 4 tiers of caches). Adding lots of memory (24GB and more) to servers helped a lot too.
Also we've made a lot of client-side improvements — "sprite" images, inlining styles and scripts, optimizing site for better browser support (yeah, should be good in IE now).
Tons of smaller improvements
We've had a long list — it is almost empty now. Time to do new things, send us your ideas!
Disclaimer: the following text is technical and can make you feel bored. Proceed at your own risk.
Servers run Linux.
Indexer is Java / C++. Indexer is built quite similar to enterprise data processing systems — pluggable services/queues/message formats. We use only utility third-party libraries (Apache Commons, etc.), all infrastructure is coded from scratch.
We get about 2 requests per second from botnets / script kiddies probing site for known vulnerabilities. We laugh at them: Ha! Ha!
Each word in your search is scanning around 24 millions of records in indexer's lexicon.
Every 30 seconds our robots: discover, download, parse, analyze encoding, normalize names, split infodata, upload to index, generate phonetic info, run SafeSearch checks, run FakeScan checks and make a new torrent available.
Also robots run around 200 jobs each day to keep systems healthy and update around 1,500 TrackerMatch records each second.
At any given moment there are at least 15 robots working on something useful. Or at least trying to look busy, you never know with robots.
Check out statistics for more details.
A Little Bit of History
Bitsnoop started as a proof of concept utility back in 2008. It was just something developed for the fun of it, running on Java and BerkeleyDB. Didn't even have a name.
It was a mere toy up to March 2009 when we've decided to start developing it on a larger scale and drew first architecture draft. This is how first live version started, code named "BitRover".
In a week we've received a review on Torrentfreak (thanks, Ernesto!) and instantly became popular — this is when it hit the fan. It was a tough week. We've done lots of performance testing — but it's a bit different when you have a horde of users on site. Tomcat was crashing with "out of memory" errors or melting half of CPUs with 100% load. Or stopping to respond for no apparent reason. Figures.
We've had to code all of our front end in PHP — in 6 days, while desperately trying to keep Tomcat alive.
Things got stable at the end of October with 2 million torrents in index — and we could start reviewing performance and planning new features. This is when first draft of "BTek3" architecture started to appear. Development took more than 3 months (well, we have quite long New Year vacation here) — and here it is, new and kicking!
We've had a catastrophic failure of our storage subsystem in February 2010 (just after St. Valentine's), which caused Bitsnoop to be offline for some time. Now everything is fixed and improved tenfold.