What Are You Doing Right Now
-
@reid-cooper said in What Are You Doing Right Now:
Have not heard of zsync, that seems pretty useful.
Actually it's used by some major Linux distros to sync ISOs between masters and mirrors.
-
@reid-cooper said in What Are You Doing Right Now:
Have not heard of zsync, that seems pretty useful.
The basic idea is that you upload your files to your webserver and generate a data block map using zsyncmake. This must be repeated every time you update a file. That's all you need on the server side.
On the client, you just invoke zsync and point it to the zsync map files (which contains the block map and the relative or absolute URL to the real file). It then compares your local version and starts to sync changed blocks.
We are syncing nearly 60GB of mostly binary files (game mods) for more than 300 highly active users. Generating the map files and a lot of other stuff (like JSON listings and hashes) takes us about 10 minutes on an average SSD.
-
@thwr said in What Are You Doing Right Now:
@scottalanmiller said in What Are You Doing Right Now:
@thwr said in What Are You Doing Right Now:
How much I love zsync. Wrote a client for our gaming community last year. Unlike rsync, there's no direct server connection beyond HTTP required.
Just synced 18.6 GB worth of updated files, took me three minutes and 20 seconds with 100 MBit downstream. Mostly disk-I/O.
Awesome for syncing files because it only transmits changed blocks. It's much more efficient than expected, saving up to 95% traffic on our bills.
Already thought about building a .NET lib. There are only Java libs available. It's not that I dislike Java. No. I hate it.
It's rsync under the hood.
-
@scottalanmiller said in What Are You Doing Right Now:
@thwr said in What Are You Doing Right Now:
@scottalanmiller said in What Are You Doing Right Now:
@thwr said in What Are You Doing Right Now:
How much I love zsync. Wrote a client for our gaming community last year. Unlike rsync, there's no direct server connection beyond HTTP required.
Just synced 18.6 GB worth of updated files, took me three minutes and 20 seconds with 100 MBit downstream. Mostly disk-I/O.
Awesome for syncing files because it only transmits changed blocks. It's much more efficient than expected, saving up to 95% traffic on our bills.
Already thought about building a .NET lib. There are only Java libs available. It's not that I dislike Java. No. I hate it.
It's rsync under the hood.
Same algorithm, yes.
-
@thwr said in What Are You Doing Right Now:
@scottalanmiller said in What Are You Doing Right Now:
@thwr said in What Are You Doing Right Now:
@scottalanmiller said in What Are You Doing Right Now:
@thwr said in What Are You Doing Right Now:
How much I love zsync. Wrote a client for our gaming community last year. Unlike rsync, there's no direct server connection beyond HTTP required.
Just synced 18.6 GB worth of updated files, took me three minutes and 20 seconds with 100 MBit downstream. Mostly disk-I/O.
Awesome for syncing files because it only transmits changed blocks. It's much more efficient than expected, saving up to 95% traffic on our bills.
Already thought about building a .NET lib. There are only Java libs available. It's not that I dislike Java. No. I hate it.
It's rsync under the hood.
Same algorithm, yes.
That's what rsync is
-
@scottalanmiller said in What Are You Doing Right Now:
@thwr said in What Are You Doing Right Now:
@scottalanmiller said in What Are You Doing Right Now:
@thwr said in What Are You Doing Right Now:
@scottalanmiller said in What Are You Doing Right Now:
@thwr said in What Are You Doing Right Now:
How much I love zsync. Wrote a client for our gaming community last year. Unlike rsync, there's no direct server connection beyond HTTP required.
Just synced 18.6 GB worth of updated files, took me three minutes and 20 seconds with 100 MBit downstream. Mostly disk-I/O.
Awesome for syncing files because it only transmits changed blocks. It's much more efficient than expected, saving up to 95% traffic on our bills.
Already thought about building a .NET lib. There are only Java libs available. It's not that I dislike Java. No. I hate it.
It's rsync under the hood.
Same algorithm, yes.
That's what rsync is
There's a saying here... "Counting peas?"
-
I did it again. I ended up using
ctrl+shift+vin Windows. -
@thwr said in What Are You Doing Right Now:
@reid-cooper said in What Are You Doing Right Now:
Have not heard of zsync, that seems pretty useful.
The basic idea is that you upload your files to your webserver and generate a data block map using zsyncmake. This must be repeated every time you update a file. That's all you need on the server side.
On the client, you just invoke zsync and point it to the zsync map files (which contains the block map and the relative or absolute URL to the real file). It then compares your local version and starts to sync changed blocks.
We are syncing nearly 60GB of mostly binary files (game mods) for more than 300 highly active users. Generating the map files and a lot of other stuff (like JSON listings and hashes) takes us about 10 minutes on an average SSD.
Are they authenticating somehow? Or does it just point to the directory?
-
@coliver said in What Are You Doing Right Now:
@thwr said in What Are You Doing Right Now:
@reid-cooper said in What Are You Doing Right Now:
Have not heard of zsync, that seems pretty useful.
The basic idea is that you upload your files to your webserver and generate a data block map using zsyncmake. This must be repeated every time you update a file. That's all you need on the server side.
On the client, you just invoke zsync and point it to the zsync map files (which contains the block map and the relative or absolute URL to the real file). It then compares your local version and starts to sync changed blocks.
We are syncing nearly 60GB of mostly binary files (game mods) for more than 300 highly active users. Generating the map files and a lot of other stuff (like JSON listings and hashes) takes us about 10 minutes on an average SSD.
Are they authenticating somehow? Or does it just point to the directory?
It's plain HTTP, so whatever your webserver can do.
There's only one downside: No HTTPS support. I do not like the idea, but that's another story. You can work around by many means: SSH port forwarding or VPN tunnels, for example.
-
@thwr said in What Are You Doing Right Now:
@coliver said in What Are You Doing Right Now:
@thwr said in What Are You Doing Right Now:
@reid-cooper said in What Are You Doing Right Now:
Have not heard of zsync, that seems pretty useful.
The basic idea is that you upload your files to your webserver and generate a data block map using zsyncmake. This must be repeated every time you update a file. That's all you need on the server side.
On the client, you just invoke zsync and point it to the zsync map files (which contains the block map and the relative or absolute URL to the real file). It then compares your local version and starts to sync changed blocks.
We are syncing nearly 60GB of mostly binary files (game mods) for more than 300 highly active users. Generating the map files and a lot of other stuff (like JSON listings and hashes) takes us about 10 minutes on an average SSD.
Are they authenticating somehow? Or does it just point to the directory?
It's plain HTTP, so whatever your webserver can do.
There's only one downside: No HTTPS support. I do not like the idea, but that's another story. You can work around by many means: SSH port forwarding or VPN tunnels, for example.
I did a small workaround, as our content (mods) is not confidential. A script puts rolling credentials into the .htaccess-file of the webservers directory, keeping only the newest X entries. Invoked via cron. A serverside webservice provides the credentials to authenticated clients. So even if someone sniffes users and passwords (both random), he could only use them for a short amount of time.
Not ideal, but like I said, it's not required to be secure. Just a small barrier.
-
Headache healed enough to allow me to play string with my cat before I head to sleep. Day = salvaged
-
Piss off Windows 7 updates temp files...
-
@jaredbusch said in What Are You Doing Right Now:
Piss off Windows 7 updates temp files...
Such a pain...
-
Just wrapped up our family game night.
-
Just ordered Sausage and Chips for my Lunch.......Hurry up dinner time!!!!
-
@hobbit666 Thanks, you've decided my lunch for me. To the chippy!
-
-
Day off today!!
-
@dbeato said in What Are You Doing Right Now:
Day off today!!
Lucky, it's getting bad outside, we're supposed to get 3-6" of snow today.
-
