chore: remove weak hashing which does not pull its weight (#10005)

We've had weak/rolling hashing in the code for quite a while. It was a
popular request for a while, based on the belief that rsync does this
and we should too. However, the benefit is quite small; we save on
average about 0.8% of transferred blocks over the population as a whole:

<img width="974" alt="Screenshot 2025-03-28 at 17 09 02"
src="https://github.com/user-attachments/assets/bbe10dea-f85e-4043-9823-7cef1220b4a2"
/>

This would be fine if the cost was comparably low, however the downside
of attempting rolling hash matching is that we (by default) do a
complete file read on the destination in order to look for matches
before we starting pulling blocks for the file. For any larger file this
means a sometimes long, I/O-intensive pause before the file starts
syncing, for usually no benefit.

I propose we simply rip off the bandaid and save the effort.

This commit is contained in:

Jakob Borg

2025-03-29 13:21:10 +01:00

committed by

GitHub

parent 1a25ae32ca

commit b1c8f88a44

38 changed files with 288 additions and 1030 deletions

									
										go.mod
									
		-1
	
												View File
												
				@@ -9,7 +9,6 @@ require (

					github.com/calmh/incontainer v1.0.0

					github.com/calmh/xdr v1.2.0

					github.com/ccding/go-stun v0.1.5

					github.com/chmduquesne/rollinghash v4.0.0+incompatible

					github.com/d4l3k/messagediff v1.2.1

					github.com/getsentry/raven-go v0.2.0

					github.com/go-ldap/ldap/v3 v3.4.10