Skip to content

Scarf Gateway

Overview

Gateway.png

Scarf Gateway is a service that sits in front of your existing software hosting platform(s), acting as a single access-point to all of your artifacts, regardless of where they are actually hosted. By making it easy to host content from your own domain, Scarf Gateway decouples your distribution from your hosting provider and provides in-depth download analytics.

Suppose you maintain a Docker container image acme/rocket-skates on Docker Hub. Your users would normally pull your container image from Docker hub directly (docker pull acme/rocket-skates.) Using Scarf, they would pull from Scarf Gateway using your private domain (docker pull docker.acme.com/acme/rocket-skates.) Important analytics are recorded and shared with you while you are using Scarf and you may change your hosting providers at any time without changing your container image name or documentation.

Your users will always be using your domain to pull your Docker container images. Your Docker container images can stay on their current hosting provider, but will be served through your own domain, e.g.:

# Make your existing container available through your own domain
$ docker pull docker.acme.com/acme/rocket-skates

# Scarf provides a domain too if you'd prefer to use ours
$ docker pull acme.docker.scarf.sh/acme/rocket-skates

Data insights about your Docker container image's downloads can be found in your Scarf Dashboard. From there, you can also manage Scarf Gateway configuration, access controls, and more.

Configuring

Scarf Package Entries

Everything that is served and tracked via Scarf Gateway needs a corresponding package entry on scarf.sh. Configuration, analytics, and permissions are all done at the level of a package, or single repository. rocket-skates, acme/rocket-skates are all valid package entries. Because packages can seamlessly change their hosting provider, hostnames (e.g. gcr.io) are not part of the package identifier on Scarf (e.g. acme/rocket-skates and not gcr.io/acme/rocket-skates.)

To create your package entry, click "New Package" in the navbar in your Scarf dashboard, or click here. Then select the corresponding package type for your artifact.

Container Image Packages

Scarf Gateway configuration for a Docker container image entry has two main considerations:

  • Backend URL: This refers to where your container is actually hosted, the location where Scarf will direct requests to pull the container. Scarf will ask for your container's current pull command. This could be rocket-skates, acme/rocket-skates (implicitly specifying Docker Hub as the hosting provider, registry-1.docker.io/acme/rocket-skates), or a fully qualified container image name docker.acme.come/acme/rocket-skates. You can modify this backend URL attribute later, and your user's Docker pulls will be instantly moved over to the new destination without having communicate anything to them.
  • Domain: This can be your own domain, or a Scarf-supplied domain, of the form <username>.docker.scarf.sh. By default your Scarf domain will be used if this field is left empty. Note that this will be part of your Docker pull command for your users. While you can update this domain, updating your public domain is a breaking change for any users on the current domain! Edit this value with caution.

If you elect to use your own domain, you'll need to add a CNAME for that domain to gateway.scarf.sh. See your DNS provider's instructions for how to add a CNAME.

See Figure 0 to see how these pieces fit together visually.

File Packages

Gateway.png

Scarf Gateway configuration for a file package entry has three main considerations:

  • Domain: Just like Docker container images, you may chose to use your own domain(s) for serving files. You may also choose to use <username>.gateway.scarf.sh provided by default by Scarf. Remember, if you elect to use your own domain, you'll need to add a CNAME for that domain to gateway.scarf.sh
  • Incoming Path: This refers to where a path on a given domain where Scarf will direct requests to fetch a file asset. This could be static path like /downloads/rocket-skates.tar.gz or a template path with variables like /files/{version}/{platform}/rocket-skates-{platform}-{version}.tar.gz. You may use variables in your incoming path as specified in https://datatracker.ietf.org/doc/html/rfc6570. You can modify a path value later, but be careful to communicate to your users because this would be a breaking change.
  • Outgoing URL: This is the full URL to your asset on your HTTP/S hosting provider. It is a template (or static) URL that may also use any variables defined in the Incoming Path. For example https://besthostingprovider.com/acme/{platform}/rocket-skates-{version}.tar.gz

See Figure 3 to see how these pieces fit together visually.

How it works

When a user requests a Docker container image through Scarf, Scarf simply issues a redirect response, pointing to whichever hosting provider you've configured for your container. Certain container runtimes do not handle redirects appropriately during authentication (which is required even for anonymous pulls), and in those cases Scarf will proxy the request to the host instead of redirecting. For a visualization of the system from the end-user's perspective, see Figure 1. For an overview of the entire system, Figure 2.

When a user requests a file through Scarf, Scarf simply issues a redirect response, pointing to whichever hosting provider you've configured for your file. For a visualization of the system from the end-user's perspective, see Figure 4. For an overview of the entire system, Figure 5.

Dashboard and Data Access

Your package's usage data will be made available to you in your Scarf dashboard. You can grant others access to your package's usage data as well from your package details page. Current permission levels supported are:

Access Level Description
Owner Can read all package-level data, edit package configuration, and grant access to other members
Admin Can read all package-level data, edit package configuration, and grant access to other members (but can't remove other admins)
Member Can read all package-level data

Scarf does not yet support organization-level permissions but will soon.

Docker Packages: Defining a container pull

Scarf defines a pull based on how Docker Hub defines them for the purposes of their rate-limiting functionality.

A pull is defined as one or more GET requests on hosting provider manifest URLs (/v2/*/manifests/*). HEAD requests are not counted as a pull.

Note that even if a client downloads the blobs that comprise any given container, the container's manifest file may already be cached on the client, meaning the download would not be counted in Scarf's analytics. Future versions of Scarf's data processing pipelines will be more intelligent and will track things like partial downloads, blob fetches, etc.

Security

All interactions through Scarf Gateway occur over HTTPS. Scarf Gateway will procure a valid TLS certificate via LetsEncrypt, and perform TLS termination for the traffic. Scarf Gateway in turn will issue a redirect for the request, or proxy the request to the hosting provider.

Availability

Scarf Gateway is a free hosted service that is provided to maintainers and users as-is and as-available.

Scarf Gateway is deployed on AWS in multiple regions around the globe; it is fault tolerant even to entire regions going offline, and can automatically and elasticly scale our backend capacity to meet whatever user traffic demands of us.

We are expecting to meet a monthly service uptime percentage of 99.9%. Guarantees of our service-level agreement will be made available in the future.

To see Scarf's uptime and system status, you can view the status page here.

Badges

All packages on Scarf Gateway offer dynamic Scarf-powered README badges automatically. Head to your package page, and the badges will be shown in the details section near the top. Copy the URL, paste it into your project’s README based on whatever doc format you are using and you’re all set.

Scarf-badge-details-page

What is the difference between the downloads badge and the company badge?

The commercial usage badge shows how many distinct companies have been identified to be fetching your Scarf Gateway package in the previous month. The downloads badge reports the total number of downloads across all users.

What is the purpose of this badge?

README badges let you show off your project by sharing high-level real-time data about your download traffic and commercial adoption, so readers can quickly assess some basic details about your project. Scarf-powered README badges are an easy way to share your project’s usage data publicly, regardless of where on the internet your docs are being rendered. Telling prospective new users how many companies use your project is a great way to show that your project is reliable and worth adopting.

What is the URL format of the badges

The badges can be used in the following formats:

  • Company Badge

    • https://scarf.sh/package/company-badge/{package-id}
    • https://scarf.sh/company-badge/{username}/{package-name}?package-type={package-type}
  • Downloads Badge

    • https://scarf.sh/package/installs-badge/{package-id}
    • https://scarf.sh/installs-badge/{username}/{package-name}?package-type={package-type}

Caveats and Limitations

A given subdomain can only point to a single container registry at a time.

If you have Docker container images on multiple distinct registries, you'll currently need to use multiple distinct subdomains (one per hosting provider). This limitation is due to the current implementation of the Docker registry authorization. To begin pulling a container, an authentication request is sent, which must be passed to the hosting provider you configure Scarf Gateway to use. Unfortunately, the initial authorization request doesn't include any information about what image it's trying to pull all Scarf Gateway has to go on is the hostname used to begin to pull the Docker container image. Subsequent Docker API requests do the actual "pulling" of an image. The core of the problem is that, if you attempt to authorize with one registry and pull an image from another, it will fail with an authorization error.

The path used in your container's new pull command must match the path on the backend container registry

If your container is on Docker Hub as acme/rocket-skates, your install command must be: docker pull ~<your-domain.com>/acme/rocket-skates. The image name path (acme/rocket-status) is not something that can be changed at this time. This is due to the Docker client's OAuth implementation (authorization includes the image name path of the being requested.) If Scarf Gateway redirects to a different path, the authorization becomes invalid and the Docker pull will fail.

FAQ

How do I get started using Scarf Gateway?

First, create an account on Scarf, if you haven’t already done so. Once you’ve registered, you’ll be prompted to create a new package. If you’re already using Scarf, you’ll be able to click “New Package” in the navigation bar.

Select “Docker” for your package type and enter in the requested details about your container. Scarf Gateway currently supports Docker containers. Support for more package and artifact types are on the way. Stay tuned.

If I use a custom domain to host my container through Scarf, what happens to my existing users? Do they all have to update?

Hosting containers on your custom domain via Scarf has no impact on your existing users; your domain adds a new path for users to download your package. You can encourage end-users to switch their pull commands over to your new domain, but they can continue pulling directly from your registry provider with no negative impact.

Should you decide to switch registries later on, current users will have to update their pull commands to either your custom domain or to the new registry URL. If they go straight to the registry, they would need to update every time you decide to switch registries. If they use your custom domain, they will never need to update it again.

Are you actually hosting my packages?

No, your package continues to be hosted on your current hosting provider not on Scarf itself. Scarf Gateway is simply a thin redirect layer in front of your provider. Since Scarf Gateway acts as a stable location on the internet for your packages, you will always have the freedom to host them with any provider you choose.

My Docker container image name on my current registry is acme/rocket-skates, can I change that to just rocket-skates when users pull through Scarf?

Unfortunately this is not possible unless you can change this name on the registry that hosts your container. Your container name on Scarf must match the container name on the registry that hosts it, because the Docker client uses that name to sign the request and validate the response from the registry. The Docker client will reject the download if the response signature is invalid. See the Caveats section for more information.

How are you managing the usage data you get about my project? Are you storing my users’ data?

Scarf Gateway does not store any personally identifying information or sensitive data about your users.

Scarf looks up IP address metadata, but the raw IP addresses are discarded and never exposed. IP metadata may contain:

  • Coarse-grained location
  • Device/OS information
  • Company information, cloud providers, etc.
  • Additionally, Scarf sees metadata about the containers that are being downloaded such as:
  • Tags/versions (variables)
  • Client runtime and version

What package types are you planning to support next?

We’d love your input to help us prioritize support for additional package types. Java, Python, and others are planned. Scarf Gateway will ultimately be generalized to support arbitrary artifact types.

How much does it cost to use Scarf Gateway?

Scarf Gateway’s current feature set is free and will remain free. We will be adding additional functionality, features, service level agreements, and more, some free and some paid.

Is Scarf Gateway self-hosted or managed by Scarf?

Scarf Gateway is managed by the Scarf team. We plan an open source release of Scarf Gateway for self-hosting, when it is out of the current open beta period and into general availability.

How long will it take for any given container download to show up in my analytics dashboard?

Downloads will typically show up in your dashboard within minutes.

Is there an API I can use to pull my stats, manage my packages, etc?

Yes! See our API documentation for more information.

I have more questions, where is the best place to ask?

Join us in Slack, we're more than happy to help.

Figures

Figure 0: Using Scarf (Docker) Gateway as a maintainer

img

Figure 1: Pulling a Docker container image from Scarf (Docker) Gateway as a User

img

Figure 2: Full System Diagram (Docker)

img

Figure 3: Using Scarf (File) Gateway as a maintainer

img

Figure 4: Downloading a File from Scarf (File) Gateway as a User

img

Figure 5: Full System Diagram (File)

img