Writing a file system in Go

bazil.org/fuse: a pure-Go FUSE library

10 June 2013

Tommi Virtanen

FUSE

File systems in User SpacE
fuse.sourceforge.net/

People tend to think it's a C library

Nope

It's a Linux kernel API

With a request-response protocol spoken over a fd

Userspace is the server

The C library is the reference implementation

... named exactly the same as the kernel API :(

All the things you'd expect from a project hosted at SourceForge in 2013.

We'll call it "C FUSE" from here on.

Others provide similar libraries

C libraries more or less API compatible with the C FUSE API.
May even be forks of it.
Lots of platform-specific extensions.

OS X:
osxfuse.github.com/
(originally code.google.com/p/macfuse/
or github.com/macfuse/macfuse)
-- MacFUSE is dead upstream, forks ahoy

Windows: dokan-dev.net/en/

hanwen/go-fuse

Han-Wen Nienhuys wrote a go-fuse library.

The structure of its API leaves a lot to be desired.

Use it if you want. Good luck.

Personally, I think writing a library from scratch would be less painful.

rsc to the rescue

Russ Cox's fuse library can be found at code.google.com/p/rsc/fuse
code.google.com/p/rsc/source/browse/#hg%2Ffuse

Bazil is a fork of it. With a name that doesn't match /fuse/.
(Strictly, Bazil is an umbrella project, bazil.org/fuse is the FUSE library.
So that was a lie. But call it "Bazil FUSE" if you need to.)

Hopefully our contributions help improve the state of the ecosystem.

Russ seems to develop mostly on OS X, Tv is a Linux person.
Both should work. OS X contributions are most welcome.

Bazil is an independent implementation

Pure-Go implementation of userspace server for the Linux and OS X kernel protocols.

Linux kernel support is upstream,
OS X needs the fusefs kernel module from OSXFUSE.

On Linux: uses setuid fusermount userspace mount helper from the C FUSE package.

Many things you read about "FUSE" are about C FUSE, and don't apply to Bazil.
For example: automatic inode numbering, multithreading.

Understanding FUSE

Kernel interface
Documentation/filesystems/fuse.txt
include/uapi/linux/fuse.h
numbered message types, wire structures, various constants

C FUSE API docs, "experimental":
fuse.sourceforge.net/doxygen/structfuse__lowlevel__ops.html
fuse.sourceforge.net/doxygen/structfuse__operations.html

Kernel API

mount(2) is given a file descriptor.

Bazil uses /bin/fusermount as subprocess (to handle /etc/mtab and
such), passes a socketpair fd to it.
source

From there on, it's just reads and writes on the fd.

Kernel protocol

RequestID:
to match response to request
lifetime ends with response

NodeID:
directory entry kernels knows about
kernel tells when to forget

HandleID:
open file
kernel tells when to destroy

Low-level C FUSE

C FUSE's low-level API exposes kernel requests and responses quite directly.
Dispatches in inode numbers, explicit response. Typical to see helper wrappers. 

static void hello_ll_getattr(fuse_req_t req, fuse_ino_t ino,
                 struct fuse_file_info *fi)
{
    struct stat stbuf;

    (void) fi;

    memset(&stbuf, 0, sizeof(stbuf));
    if (hello_stat(ino, &stbuf) == -1)
        fuse_reply_err(req, ENOENT);
    else
        fuse_reply_attr(req, &stbuf, 1.0);
}

source

Low-level C FUSE, 2

static int hello_stat(fuse_ino_t ino, struct stat *stbuf)
{
    stbuf->st_ino = ino;
    switch (ino) {
    case 1:
        stbuf->st_mode = S_IFDIR | 0755;
        stbuf->st_nlink = 2;
        break;

    case 2:
        stbuf->st_mode = S_IFREG | 0444;
        stbuf->st_nlink = 1;
        stbuf->st_size = strlen(hello_str);
        break;

    default:
        return -1;
    }
    return 0;
}

source

High-level C FUSE

Dispatches on path names instead of inode numbers.
Response is implicit. Less need for helper wrappers. 

static int hello_getattr(const char *path, struct stat *stbuf)
{
    int res = 0;

    memset(stbuf, 0, sizeof(struct stat));
    if (strcmp(path, "/") == 0) {
        stbuf->st_mode = S_IFDIR | 0755;
        stbuf->st_nlink = 2;
    } else if (strcmp(path, hello_path) == 0) {
        stbuf->st_mode = S_IFREG | 0444;
        stbuf->st_nlink = 1;
        stbuf->st_size = strlen(hello_str);
    } else
        res = -ENOENT;

    return res;
}

source

Bazil API

Bazil doesn't do either inode number or path dispatching.

Instead, it mirrors kernel data structures, with uint64 handles to objects.

It's also significantly closer to the wire protocol.

Requests are served by methods on the node itself, not a global dispatch function.

Feel free to construct an in-memory tree, for simple filesystems.

Interfaces

A Node must implement Attr(). Everything else is optional.

For example, if a node does not implement Remove(), unlink(2) on it will always fail.

Documentation of possible interfaces is severely lacking right now.
Need to explore how to maintain it in sync with the code.

Directory lookup in Bazil

Kernel struct dentry maps to fuse.Node, identified on wire with fuse.NodeID.

Lookup() returns a Node, a reference is kept in a map[NodeID]Node until a Forget() call

Open files in Bazil

Open file: kernel struct file maps to fuse.Handle, identified on wire with fuse.HandleID.

Open() returns a Handle (maybe self), which is kept in a map[HandleID]Handle until a Destroy() call.

close(2) has two parts:
per fd method Release() that returns error (for delayed writes and such),
and a final Destroy() that always succeeds.

Let's Go!

type File struct{}

func (File) Attr() fuse.Attr {
    return fuse.Attr{Mode: 0444}
}

File content

func (File) ReadAll(intr fuse.Intr) ([]byte, fuse.Error) {
    return []byte("hello, world\n"), nil
}

ReadAll() caches the whole content in memory and serves smaller reads from that.

It's convenient for pseudofiles.

There's also Read() for more realistic use.

Looking up the file by name

type Dir struct{}

func (Dir) Attr() fuse.Attr {
    return fuse.Attr{Inode: 1, Mode: os.ModeDir | 0555}
}

func (Dir) Lookup(name string, intr fuse.Intr) (fuse.Node, fuse.Error) {
    if name == "hello" {
        return File{}, nil
    }
    return nil, fuse.ENOENT
}

Starting point

type FS struct{}

func (FS) Root() (fuse.Node, fuse.Error) {
    return Dir{}, nil
}

Listing files

var dirDirs = []fuse.Dirent{
    {Inode: 2, Name: "hello", Type: fuse.DT_File},
}

func (Dir) ReadDir(intr fuse.Intr) ([]fuse.Dirent, fuse.Error) {
    return dirDirs, nil
}

Boilerplate

// Copyright 2012 The Go Authors.  All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

// Hellofs implements a simple "hello world" file system.
package main

import (
    "flag"
    "fmt"
    "log"
    "os"

    "bazil.org/fuse"
)

var Usage = func() {
    fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0])
    fmt.Fprintf(os.Stderr, "  %s MOUNTPOINT\n", os.Args[0])
    flag.PrintDefaults()
}

...and Go!

func main() {
    flag.Usage = Usage
    flag.Parse()

    if flag.NArg() != 1 {
        Usage()
        os.Exit(2)
    }
    mountpoint := flag.Arg(0)

    c, err := fuse.Mount(mountpoint)
    if err != nil {
        log.Fatal(err)
    }

    c.Serve(FS{})
}

Bazil internals

API decisions in bazil.org/fuse

Special Error type

Bazil methods return fuse.Error and not Go's usual error.

It has hidden knowledge of what POSIX errno to return to kernel.

Use one of fuse.EIO, fuse.EPERM, fuse.ENOENT, fuse.ENOSYS, fuse.ESTALE, etc.

Worst case, use fuse.Errno(syscall.ENOTDIR) etc.
Ideally, Bazil will provide ready errors for all those.

Inode numbers

If you don't fill in inode numbers in your Attr() etc calls, Bazil will hash the full path to create a pseudorandom inode number.

These may collide, causing confusion in low-level tools like find.

This also currently forces Bazil to remember full pathname to every Node.
Details are likely to change, if a cheaper replacement scheme can be figured out.

If you care, manage inode numbers explicitly.

Concurrency

    for {
        req, err := c.ReadRequest()
        if err != nil {
            if err == io.EOF {
                break
            }
            return err
        }

        go c.serve(fs, req)
    }

Each request is served in a separate goroutine.

Sort of like net/http.

If you mutate shared data, use mutexes.

Bazillion Bytes

bazil.org/

FUSE can be FUN

Go do it

Thank you

Tommi Virtanen

[email protected]

@tv