github.com/TRON-US/go-btfs

Ipfs 0.4.11 is a larger release that brings many long-awaited features and performance improvements. These include new datastore options, more efficient bitswap transfers, greatly improved resource consumption, circuit relay support, ipld plugins, and more! Take a look at the full changelog below for a detailed list of every change.

The ipfs datastore has, until now, been a combination of leveldb and a custom git-like storage backend called 'flatfs'. This works well enough for the average user, but different ipfs usecases demand different backend configurations. To address this, we have changed the configuration file format for datastores to be a modular way of specifying exactly how you want the datastore to be structured. You will now be able to configure ipfs to use flatfs, leveldb, badger, an in-memory datastore, and more to suit your needs. See the new datastore documentation for more information.

Bitswap received some much needed attention during this release cycle. The concept of 'Bitswap Sessions' allows bitswap to associate requests for different blocks to the same underlying session, and from that infer better ways of requesting that data. In more concrete terms, parts of the ipfs codebase that take advantage of sessions (currently, only ipfs pin add) will cause much less extra traffic than before. This is done by making optimistic guesses about which nodes might be providing given blocks and not sending wantlist updates to every connected bitswap partner, as well as searching the DHT for providers less frequently. In future releases we will migrate over more ipfs commands to take advantage of bitswap sessions. As nodes update to this and future versions, expect to see idle bandwidth usage on the ipfs network go down noticeably.

The never ending effort to reduce resource consumption had a few important updates this release. First, the bitswap sessions changes discussed above will help with improving bandwidth usage. Aside from that there are two important libp2p updates that improved things significantly. The first was a fix to a bug in the dial limiter code that was causing it to not limit outgoing dials correctly. This resulted in ipfs running out of file descriptors very frequently (as well as incurring a decent amount of excess outgoing bandwidth), this has now been fixed. Users who previously received "too many open files" errors should see this much less often in 0.4.11. The second change was a memory leak in the DHT that was identified and fixed. Streams being tracked in a map in the DHT weren't being cleaned up after the peer disconnected leading to the multiplexer session not being cleaned up properly. This issue has been resolved, and now memory usage appears to be stable over time. There is still a lot of work to be done improving memory usage, but we feel this is a solid victory.

It is often said that NAT traversal is the hardest problem in peer to peer technology, we tend to agree with this. In an effort to provide a more ubiquitous p2p mesh, we have implemented a relay mechanism that allows willing peers to relay traffic for other peers who might not otherwise be able to communicate with each other. This feature is still pretty early, and currently users have to manually connect through a relay. The next step in this endeavour is automatic relaying, and research for this is currently in progress. We expect that when it lands, it will improve the perceived performance of ipfs by spending less time attempting connections to hard to reach nodes. A short guide on using the circuit relay feature can be found here.

The last feature we want to highlight (but by no means the last feature in this release) is our new plugin system. There are many different workflows and usecases that ipfs should be able to support, but not everyone wants to be able to use every feature. We could simply merge in all these features, but that causes problems for several reasons: first off, the size of the ipfs binary starts to get very large very quickly. Second, each of these different pieces needs to be maintained and updated independently, which would cause significant churn in the codebase. To address this, we have come up with a system that allows users to install plugins to the vanilla ipfs daemon that augment its capabilities. The first of these plugins are a git plugin that allows ipfs to natively address git objects and an ethereum plugin that lets ipfs ingest and operate on all ethereum blockchain data. Soon to come are plugins for the bitcoin and zcash data formats. In the future, we will be adding plugins for other things like datastore backends and specialized libp2p network transports. You can read more on this topic in Plugin docs

In order to simplify its integration with fs-repo-migrations, we've switched the ipfs/go-ipfs docker image from a musl base to a glibc base. For most users this will not be noticeable, but if you've been building your own images based off this image, you'll have to update your dockerfile. We recommend a multi-stage dockerfile, where the build stage is based off of a regular Debian or other glibc-based image, and the assembly stage is based off of the ipfs/go-ipfs image, and you copy build artifacts from the build stage to the assembly stage. Note, if you are using the docker image and see a deprecation message, please update your usage. We will stop supporting the old method of starting the dockerfile in the next release.

Finally, I would like to thank all of our contributors, users, supporters, and friends for helping us along the way. Ipfs would not be where it is without you.

• Features
  • Add --pin option to ipfs dag put (ipfs/go-ipfs#4004)
  • Add --pin option to ipfs object put (ipfs/go-ipfs#4095)
  • Implement --profile option on ipfs init (ipfs/go-ipfs#4001)
  • Add CID Codecs to ipfs block put (ipfs/go-ipfs#4022)
  • Bitswap sessions (ipfs/go-ipfs#3867)
  • Create plugin API and loader, add ipld-git plugin (ipfs/go-ipfs#4033)
  • Make announced swarm addresses configurable (ipfs/go-ipfs#3948)
  • Reprovider strategies (ipfs/go-ipfs#4113)
  • Circuit Relay integration (ipfs/go-ipfs#4091)
  • More configurable datastore configs (ipfs/go-ipfs#3575)
  • Add experimental support for badger datastore (ipfs/go-ipfs#4007)
• Improvements
  • Add better support for Raw Nodes in MFS and elsewhere (ipfs/go-ipfs#3996)
  • Added file size to response of ipfs add command (ipfs/go-ipfs#4082)
  • Add /dnsaddr bootstrap nodes (ipfs/go-ipfs#4127)
  • Do not publish public keys extractable from ID (ipfs/go-ipfs#4020)
• Documentation
  • Adding documentation that PubSub Sub can be encoded. (ipfs/go-ipfs#3909)
  • Add Comms items from js-ipfs, including blog (ipfs/go-ipfs#3936)
  • Add Developer Certificate of Origin (ipfs/go-ipfs#4006)
  • Add transports.md document (ipfs/go-ipfs#4034)
  • Add experimental-features.md document (ipfs/go-ipfs#4036)
  • Update release docs (ipfs/go-ipfs#4165)
  • Add documentation for datastore configs (ipfs/go-ipfs#4223)
  • General update and clean-up of docs (ipfs/go-ipfs#4222)
• Bugfixes
  • Fix shutdown check in t0023 (ipfs/go-ipfs#3969)
  • Fix pinning of unixfs sharded directories (ipfs/go-ipfs#3975)
  • Show escaped url in gateway 404 message (ipfs/go-ipfs#4005)
  • Fix early opening of bitswap message sender (ipfs/go-ipfs#4069)
  • Fix determination of 'root' node in dag put (ipfs/go-ipfs#4072)
  • Fix bad multipart message panic in gateway (ipfs/go-ipfs#4053)
  • Add blocks to the blockstore before returning them from blockservice sessions (ipfs/go-ipfs#4169)
  • Various fixes for /ipfs fuse code (ipfs/go-ipfs#4194)
  • Fix memory leak in dht stream tracking (ipfs/go-ipfs#4251)
• General Changes and Refactorings
  • Require go 1.8 (ipfs/go-ipfs#4044)
  • Change IPFS to use the new pluggable Block to IPLD decoding framework. (ipfs/go-ipfs#4060)
  • Remove tour command from ipfs (ipfs/go-ipfs#4123)
  • Add support for Go 1.9 (ipfs/go-ipfs#4156)
  • Remove some dead code (ipfs/go-ipfs#4204)
  • Switch docker image from musl to glibc (ipfs/go-ipfs#4219)

Ipfs 0.4.10 is a patch release that contains several exciting new features, bugfixes and general improvements. Including new commands, easier corruption recovery, and a generally cleaner codebase.

The ipfs pin command has two new subcommands, verify and update. ipfs pin verify is used to scan the repo for pinned object graphs and check their integrity. Any issues are reported back with helpful error text to make error recovery simpler. This subcommand was added to help recover from datastore corruptions, particularly if using the experimental filestore and accidentally deleting tracked files. ipfs pin update was added to make the task of keeping a large, frequently changing object graph pinned. Previously users had to call ipfs pin rm on the old pin, and ipfs pin add on the new one. The 'new' ipfs pin add call would be very expensive as it would need to verify the entirety of the graph again. The ipfs pin update command takes shortcuts, portions of the graph that were covered under the old pin are assumed to be fine, and the command skips checking them.

Next up, we have finally implemented an ipfs shutdown command so users can shut down their ipfs daemons via the API. This is especially useful on platforms that make it difficult to control processes (Android, for example), and is also useful when needing to shut down a node remotely and you do not have access to the machine itself.

ipfs add has gained a new flag; the --hash flag allows you to select which hash function to use and we have given it the ability to select blake2b-256. This pushes us one step closer to shifting over to using blake2b as the default. Blake2b is significantly faster than sha2-256, and also is conjectured to provide superior security.

We have also finally implemented a very early (and experimental) ipfs p2p. This command and its subcommands will allow you to open up arbitrary streams to other ipfs peers through libp2p. The interfaces are a little bit clunky right now, but shouldn't get in the way of anyone wanting to try building a fully peer to peer application on top of ipfs and libp2p. For more info on this command, to ask questions, or to provide feedback, head over to the feedback issue for the command.

A few other subcommands and flags were added around the API, as well as many other requested improvements. See below for the full list of changes.

• Features
  • Add support for specifying the hash function in ipfs add (ipfs/go-ipfs#3919)
  • Implement ipfs key {rm, rename} (ipfs/go-ipfs#3892)
  • Implement ipfs shutdown command (ipfs/go-ipfs#3884)
  • Implement ipfs pin update (ipfs/go-ipfs#3846)
  • Implement ipfs pin verify (ipfs/go-ipfs#3843)
  • Implemented experimental p2p commands (ipfs/go-ipfs#3943)
• Improvements
  • Add MaxStorage field to output of "repo stat" (ipfs/go-ipfs#3915)
  • Add Suborigin header to gateway responses (ipfs/go-ipfs#3914)
  • Add "--file-order" option to "filestore ls" and "verify" (ipfs/go-ipfs#3938)
  • Allow selecting ipns keys by Peer ID (ipfs/go-ipfs#3882)
  • Don't redirect to trailing slash in gateway for go get (ipfs/go-ipfs#3963)
  • Add 'ipfs dht findprovs --num-providers' to allow choosing number of providers to find (ipfs/go-ipfs#3966)
  • Make sure all keystore keys get republished (ipfs/go-ipfs#3951)
• Documentation
  • Adding documentation on PubSub encodings (ipfs/go-ipfs#3909)
  • Change 'neccessary' to 'necessary' (ipfs/go-ipfs#3941)
  • README.md: add Nix to the linux package managers (ipfs/go-ipfs#3939)
  • More verbose errors in filestore (ipfs/go-ipfs#3964)
• Bugfixes
  • Fix typo in message when file size check fails (ipfs/go-ipfs#3895)
  • Clean up bitswap ledgers when disconnecting (ipfs/go-ipfs#3437)
  • Make odds of 'process added after close' panic less likely (ipfs/go-ipfs#3940)
• General Changes and Refactorings
  • Remove 'ipfs diag net' from codebase (ipfs/go-ipfs#3916)
  • Update to dht code with provide announce option (ipfs/go-ipfs#3928)
  • Apply the megacheck code vetting tool (ipfs/go-ipfs#3949)
  • Expose port 8081 in docker container for /ws listener (ipfs/go-ipfs#3954)