Crate iroh_blobs

Source
Expand description

§iroh-blobs

This crate provides blob and collection transfer support for iroh. It implements a simple request-response protocol based on blake3 verified streaming.

A request describes data in terms of blake3 hashes and byte ranges. It is possible to request blobs or ranges of blobs, as well as collections.

The requester opens a quic stream to the provider and sends the request. The provider answers with the requested data, encoded as blake3 verified streams, on the same quic stream.

This crate is usually used together with iroh, but can also be used with normal quinn connections. Connection establishment is left up to the user or a higher level APIs such as the iroh CLI.

§Concepts

  • Blob: a sequence of bytes of arbitrary size, without any metadata.

  • Link: a 32 byte blake3 hash of a blob.

  • HashSeq: a blob that contains a sequence of links. Its size is a multiple of 32.

  • Provider: The side that provides data and answers requests. Providers wait for incoming requests from Requests.

  • Requester: The side that asks for data. It is initiating requests to one or many providers.

§Getting started

The iroh-blobs protocol was designed to be used in conjunction with iroh. Iroh is a networking library for making direct connections, these connections are what power the data transfers in iroh-blobs.

Iroh provides a Router that takes an Endpoint and any protocols needed for the application. Similar to a router in webserver library, it runs a loop accepting incoming connections and routes them to the specific protocol handler, based on ALPN.

Here is a basic example of how to set up iroh-blobs with iroh:

use iroh::{protocol::Router, Endpoint};
use iroh_blobs::{net_protocol::Blobs, util::local_pool::LocalPool};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // create an iroh endpoint that includes the standard discovery mechanisms
    // we've built at number0
    let endpoint = Endpoint::builder().discovery_n0().bind().await?;

    // spawn a local pool with one thread per CPU
    // for a single threaded pool use `LocalPool::single`
    let local_pool = LocalPool::default();

    // create an in-memory blob store
    // use `iroh_blobs::net_protocol::Blobs::persistent` to load or create a
    // persistent blob store from a path
    let blobs = Blobs::memory().build(local_pool.handle(), &endpoint);

    // turn on the "rpc" feature if you need to create blobs and tags clients
    let blobs_client = blobs.client();
    let tags_client = blobs_client.tags();

    // build the router
    let router = Router::builder(endpoint)
        .accept(iroh_blobs::ALPN, blobs.clone())
        .spawn()
        .await?;

    // do fun stuff with the blobs protocol!
    // make sure not to drop the local_pool before you are finished
    router.shutdown().await?;
    drop(local_pool);
    drop(tags_client);
    Ok(())
}

§Examples

Examples that use iroh-blobs can be found in the iroh crate. the iroh crate publishes iroh_blobs as iroh::bytes.

§License

This project is licensed under either of

at your option.

§Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Blobs layer for iroh.

The crate is designed to be used from the iroh crate, which provides a high level interface, but can also be used standalone.

It implements a protocol for streaming content-addressed data transfer using BLAKE3 verified streaming.

It also provides a store interface for storage of blobs and outboards, as well as a persistent and a memory store implementation.

To implement a server, the provider module provides helpers for handling connections and individual requests given a store.

To perform get requests, the get module provides utilities to perform requests and store the result in a store, as well as a low level state machine for executing requests.

The downloader module provides a component to download blobs from multiple sources and store them in a store.

Re-exports§

  • pub use crate::util::Tag;
  • pub use crate::util::TempTag;

Modules§

  • clicli
    Define blob-related commands.
  • downloaderdownloader
    Handle downloading blobs and collections concurrently and from nodes.
  • export
    Functions to export data from a store
  • format
    Defines data formats for HashSeq.
  • get
    The client side API
  • hashseq
    traits related to collections of blobs
  • metrics
    Metrics for iroh-blobs
  • net_protocolnet_protocol
    Adaptation of iroh-blobs as an iroh protocol.
  • protocol
    Protocol for transferring content-addressed blobs and collections over quic connections. This can be used either with normal quic connections when using the quinn crate or with magicsock connections when using the iroh-net crate.
  • provider
    The server side API
  • rpcrpc
    Provides a rpc protocol as well as a client for the protocol
  • store
    Implementations of blob stores
  • ticket
    Tickets for blobs.
  • util
    Utility functions and types.

Structs§

Enums§

Constants§

  • ALPN
    The ALPN used with quic for the iroh bytes protocol.
  • IROH_BLOCK_SIZE
    Block size used by iroh, 2^4*1024 = 16KiB