Multithreaded file hashing and de-duplication.

So I hacked away for a few hours and now have a working multi threaded up loader. It currently has some errors in some of the logic I haven’t had time to debug yet.
But it works flawless so-far in syncing my source code directories on my FreeBSD box and my Linux Pinebook. The best part is the inbuilt de-duplication which uses hashing to determine file signatures and upload or download them based on changes.
This will soon have a diff feature to compare the differences in files.
Anyway I hope it helps. All files are stored on my Digital Ocean droplet so I can access them when away from home. De-duplication means I only download changed files (usually only a few kB) which is great for working on the road.
I want to move for HTTPS for speed because the SSH overhead is terrible.

Leave a comment

Your email address will not be published. Required fields are marked *