Conversation

Notices

1. Nate Cull (natecull@mastodon.social)'s status on Tuesday, 02-Jan-2018 20:41:55 EST Nate Cull

   Why I Don't Think Content Addressing Is The Answer

   There's a lot of interest in content-addressed distributed stores today, where you punch in a hash of your content (a file) and get that file back.

   tldr: Superficially this looks nice, but I think this is a bad idea and we will deeply regret it if we go this route.

   The reason: you need BOTH a hash AND a formal 'location' . The hash is there to warn you if the document has been altered in transit.

   It can't do that if it's ALSO the location!

   1. clacke@libranet.de ❌ (notclacke@pleroma.soykaf.com)'s status on Tuesday, 02-Jan-2018 22:25:49 EST clacke@libranet.de ❌

      in reply to
      @natecull Meh.
      
      http://wiki.c2.com/?WillyWonkaConflictResolution , it works, mate.
      
      You get the occasional unmitigated disaster and then you move on. 256 is a boatload of bits, Should Be Enough For Anyone, birthday paradox or not.
      
      And as someone mentioned, as soon as someone feels that a 256-bit planetary content store threatens the fabric of society, we just swap in a 512-bit hash instead, and the next cataclysmic collision will happen after the heat death of the universe.
      1. lnxw48a1 (lnxw48a1@nu.federati.net)'s status on Wednesday, 03-Jan-2018 00:09:36 EST lnxw48a1

         in reply to
         @notclacke @clacke You have to realize that @natecull is looking for a grand unified theory that describes data locally and remotely, at embedded and enterprise cloud scales and everything in between. A worthwhile exploration.
         Secondly, realize that IPFS data in theory lasts forever. You probably don't want your banking documents and family photos damaged or destroyed by random collisions (which are entirely predictable and expected). Before you say you wouldn't put them there, I saw a startup that hopes to use IPFS to host its B2B mesh computing platform. Your employer may put its (and your) info there.
         He's right. They're building a bridge, but the suspension cables are attached to the muddy riverbank, not solid ground.
         1. clacke (clacke@social.heldscal.la)'s status on Wednesday, 03-Jan-2018 01:26:12 EST clacke

            in reply to
            @lnxw48a1 @natecull
            
            > damaged or destroyed by random collisions (which are entirely predictable and expected).
            
            But they're not predictable and expected. If we encode all the atoms in the universe, 512 bits is still a big deal. I wasn't actually exaggerating when I said the heat death of the universe, although human civilization will probably have gone through several grand data theories and ultimately perished long before that happens.
            
            Github allegedly had some collisions with 128 bits in a big shared git storage thing, but it's unclear if that really happened or if it was a hypothesis for what later turned out to be normal computer crap happening. Anyway, thankfully, git people are already doing the footwork to migrate to 256 bits.
            
            256 bits isn't twice as good as 128 bits, it's 2^128 times as good. That's in the ballpark of 3*10^38.
            
            The birthday paradox is a huge deal when you're dealing with small, linear numbers, but in these terms we're already in exponential space: It eats about half your bits. That's why I think when we get our first meaningful collision on 256 bits, we'll just move to 512 bits and that will be the end of the story.
         2. Nate Cull (natecull@mastodon.social)'s status on Wednesday, 03-Jan-2018 00:14:01 EST Nate Cull

            in reply to

            @lnxw48a1 @notclacke @clacke

            Right. The use case I'm thinking of is basically in the ballpark of Named Data Networking - either analogues of it, or what fediverse-type systems could run over it if NDN gets off the ground and we think it's a sufficient basis for, eg, stuff like Scuttlebutt.

            https://en.wikipedia.org/wiki/Named_data_networking

            https://named-data.net/

            NDN is not content-addressed but uses path strings (and returns packets, I think)

            1. clacke (clacke@social.heldscal.la)'s status on Wednesday, 03-Jan-2018 01:37:34 EST clacke

               in reply to
               @natecull @lnxw48a1 I skimmed the article, but it looks like it could be described in a few ways:
               
               Like IPFS, it addresses data structure rather than network structure, but unlike IPFS, it has human-chosen names for things rather than hashes.
               
               Like the web, it has an authority structure where someone controls a name and what's published under that name, but unlike the web, this naming structure is uncoupled from the location and connection of physical nodes in the network, it's just about the data.
               
               From my quick reading, it seems to address an interactive web, not just a CDN, but I didn't get a clear idea of the mechanisms. Maybe I'll read it properly at a later time.
               
               Sounds like interesting stuff! And for some applications we probably want it. But I think the opportunistic deduplication in CANs will make them relevant and beneficial for many types of applications that today use the web.