Merge pull request #305 from jzelinskie/ext

Refactor all extendable code into ext/

Documentation/api_v1.md

@@ -19,7 +19,7 @@
   - [DELETE](#delete-namespacesnsnamevulnerabilitiesvulnnamefixesfeaturename)
 - [Notifications](#notifications)
   - [GET](#get-notificationsname)
-  - [DELETE](#delete-notificationname)
+  - [DELETE](#delete-notificationsname)
 
 ## Error Handling
 

README.md

@@ -14,15 +14,15 @@ Please use [releases] instead of the `master` branch in order to get stable bina
 Clair is an open source project for the static analysis of vulnerabilities in [appc] and [docker] containers.
 
 Vulnerability data is continuously imported from a known set of sources and correlated with the indexed contents of container images in order to produce lists of vulnerabilities that threaten a container.
-When vulnerability data changes upstream, the previous state and new state of the vulnerability along with the images they affect can be sent via webhook to a configured endpoint.
+When vulnerability data changes upstream, a notification can be delivered, and the API queried to provide the previous state and new state of the vulnerability along with the images affected by both.
-All major components can be [customized programmatically] at compile-time without forking the project.
+All major components can be [extended programmatically] at compile-time without forking the project.
 
 Our goal is to enable a more transparent view of the security of container-based infrastructure.
 Thus, the project was named `Clair` after the French term which translates to *clear*, *bright*, *transparent*.
 
 [appc]: https://github.com/appc/spec
 [docker]: https://github.com/docker/docker/blob/master/image/spec/v1.md
-[customized programmatically]: #customization
+[extended programmatically]: #customization
 [releases]: https://github.com/coreos/clair/releases
 
 ## Common Use Cases
@@ -127,7 +127,8 @@ While container images for every releases are available at [quay.io/repository/c
 
 ## Documentation
 
-The latest stable documentation can be found [on the CoreOS website]. Documentation for the current branch can be found [inside the Documentation directory][docs-dir] at the root of the project's source code.
+The latest stable documentation can be found [on the CoreOS website].
+Documentation for the current branch can be found [inside the Documentation directory][docs-dir] at the root of the project's source code.
 
 [on the CoreOS website]: https://coreos.com/clair/docs/latest/
 [docs-dir]: /Documentation
@@ -140,10 +141,10 @@ The latest stable documentation can be found [on the CoreOS website]. Documentat
 
 - *Image* - a tarball of the contents of a container
 - *Layer* - an *appc* or *Docker* image that may or maybe not be dependent on another image
-- *Detector* - a Go package that identifies the content, *namespaces* and *features* from a *layer*
-- *Namespace* - a context around *features* and *vulnerabilities* (e.g. an operating system)
 - *Feature* - anything that when present could be an indication of a *vulnerability* (e.g. the presence of a file or an installed software package)
-- *Fetcher* - a Go package that tracks an upstream vulnerability database and imports them into Clair
+- *Feature Namespace* - a context around *features* and *vulnerabilities* (e.g. an operating system)
+- *Vulnerability Updater* - a Go package that tracks upstream vulnerability data and imports them into Clair
+- *Vulnerability Metadata Appender* - a Go package that tracks upstream vulnerability metadata and appends them into vulnerabilities managed by Clair
 
 ### Vulnerability Analysis
 
@@ -164,13 +165,13 @@ By indexing the features of an image into the database, images only need to be r
 | [Red Hat Security Data]       | CentOS 5, 6, 7 namespaces                                                | [rpm]  | [CVRF]          |
 | [Oracle Linux Security Data]  | Oracle Linux 5, 6, 7 namespaces                                          | [rpm]  | [CVRF]          |
 | [Alpine SecDB]                | Alpine 3.3, Alpine 3.4 namespaces                                        | [apk]  | [MIT]           |
-| [NVD]                         | Generic Vulnerability Metadata                                           | N/A    | [Public Domain] |
+| [NIST NVD]                    | Generic Vulnerability Metadata                                           | N/A    | [Public Domain] |
 
 [Debian Security Bug Tracker]: https://security-tracker.debian.org/tracker
 [Ubuntu CVE Tracker]: https://launchpad.net/ubuntu-cve-tracker
 [Red Hat Security Data]: https://www.redhat.com/security/data/metrics
 [Oracle Linux Security Data]: https://linux.oracle.com/security/
-[NVD]: https://nvd.nist.gov
+[NIST NVD]: https://nvd.nist.gov
 [dpkg]: https://en.wikipedia.org/wiki/dpkg
 [rpm]: http://www.rpm.org
 [Debian]: https://www.debian.org/license
@@ -185,21 +186,13 @@ By indexing the features of an image into the database, images only need to be r
 ### Customization
 
 The major components of Clair are all programmatically extensible in the same way Go's standard [database/sql] package is extensible.
+Everything extendable is located in the `ext` directory.
 
-Custom behavior can be accomplished by creating a package that contains a type that implements an interface declared in Clair and registering that interface in [init()]. To expose the new behavior, unqualified imports to the package must be added in your [main.go], which should then start Clair using `Boot(*config.Config)`.
+Custom behavior can be accomplished by creating a package that contains a type that implements an interface declared in Clair and registering that interface in [init()].
+To expose the new behavior, unqualified imports to the package must be added in your own custom [main.go], which should then start Clair using `Boot(*config.Config)`.
 
-The following interfaces can have custom implementations registered via [init()] at compile time:
-
-- `Datastore` - the backing storage
-- `Notifier` - the means by which endpoints are notified of vulnerability changes
-- `Fetcher` - the sources of vulnerability data that is automatically imported
-- `MetadataFetcher` - the sources of vulnerability metadata that is automatically added to known vulnerabilities
-- `DataDetector` - the means by which contents of an image are detected
-- `FeatureDetector` - the means by which features are identified from a layer
-- `NamespaceDetector` - the means by which a namespace is identified from a layer
-
-[init()]: https://golang.org/doc/effective_go.html#init
 [database/sql]: https://godoc.org/database/sql
+[init()]: https://golang.org/doc/effective_go.html#init
 [main.go]: https://github.com/coreos/clair/blob/master/cmd/clair/main.go
 
 ## Related Links

api/api.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -23,29 +23,37 @@ import (
 	"strconv"
 	"time"
 
+	"github.com/coreos/pkg/capnslog"
 	"github.com/tylerb/graceful"
 
-	"github.com/coreos/clair/api/context"
+	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/config"
+	"github.com/coreos/clair/pkg/stopper"
-	"github.com/coreos/clair/utils"
-	"github.com/coreos/pkg/capnslog"
 )
 
 const timeoutResponse = `{"Error":{"Message":"Clair failed to respond within the configured timeout window.","Type":"Timeout"}}`
 
 var log = capnslog.NewPackageLogger("github.com/coreos/clair", "api")
 
-func Run(config *config.APIConfig, ctx *context.RouteContext, st *utils.Stopper) {
+// Config is the configuration for the API service.
+type Config struct {
+	Port                      int
+	HealthPort                int
+	Timeout                   time.Duration
+	PaginationKey             string
+	CertFile, KeyFile, CAFile string
+}
+
+func Run(cfg *Config, store database.Datastore, st *stopper.Stopper) {
 	defer st.End()
 
 	// Do not run the API service if there is no config.
-	if config == nil {
+	if cfg == nil {
 		log.Infof("main API service is disabled.")
 		return
 	}
-	log.Infof("starting main API on port %d.", config.Port)
+	log.Infof("starting main API on port %d.", cfg.Port)
 
-	tlsConfig, err := tlsClientConfig(config.CAFile)
+	tlsConfig, err := tlsClientConfig(cfg.CAFile)
 	if err != nil {
 		log.Fatalf("could not initialize client cert authentication: %s\n", err)
 	}
@@ -57,33 +65,33 @@ func Run(config *config.APIConfig, ctx *context.RouteContext, st *utils.Stopper)
 		Timeout:          0,    // Already handled by our TimeOut middleware
 		NoSignalHandling: true, // We want to use our own Stopper
 		Server: &http.Server{
-			Addr:      ":" + strconv.Itoa(config.Port),
+			Addr:      ":" + strconv.Itoa(cfg.Port),
 			TLSConfig: tlsConfig,
-			Handler:   http.TimeoutHandler(newAPIHandler(ctx), config.Timeout, timeoutResponse),
+			Handler:   http.TimeoutHandler(newAPIHandler(cfg, store), cfg.Timeout, timeoutResponse),
 		},
 	}
 
-	listenAndServeWithStopper(srv, st, config.CertFile, config.KeyFile)
+	listenAndServeWithStopper(srv, st, cfg.CertFile, cfg.KeyFile)
 
 	log.Info("main API stopped")
 }
 
-func RunHealth(config *config.APIConfig, ctx *context.RouteContext, st *utils.Stopper) {
+func RunHealth(cfg *Config, store database.Datastore, st *stopper.Stopper) {
 	defer st.End()
 
 	// Do not run the API service if there is no config.
-	if config == nil {
+	if cfg == nil {
 		log.Infof("health API service is disabled.")
 		return
 	}
-	log.Infof("starting health API on port %d.", config.HealthPort)
+	log.Infof("starting health API on port %d.", cfg.HealthPort)
 
 	srv := &graceful.Server{
 		Timeout:          10 * time.Second, // Interrupt health checks when stopping
 		NoSignalHandling: true,             // We want to use our own Stopper
 		Server: &http.Server{
-			Addr:    ":" + strconv.Itoa(config.HealthPort),
+			Addr:    ":" + strconv.Itoa(cfg.HealthPort),
-			Handler: http.TimeoutHandler(newHealthHandler(ctx), config.Timeout, timeoutResponse),
+			Handler: http.TimeoutHandler(newHealthHandler(store), cfg.Timeout, timeoutResponse),
 		},
 	}
 
@@ -94,8 +102,8 @@ func RunHealth(config *config.APIConfig, ctx *context.RouteContext, st *utils.St
 
 // listenAndServeWithStopper wraps graceful.Server's
 // ListenAndServe/ListenAndServeTLS and adds the ability to interrupt them with
-// the provided utils.Stopper
+// the provided stopper.Stopper.
-func listenAndServeWithStopper(srv *graceful.Server, st *utils.Stopper, certFile, keyFile string) {
+func listenAndServeWithStopper(srv *graceful.Server, st *stopper.Stopper, certFile, keyFile string) {
 	go func() {
 		<-st.Chan()
 		srv.Stop(0)

api/context/context.go

@@ -1,64 +0,0 @@
-// Copyright 2015 clair authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package context
-
-import (
-	"net/http"
-	"strconv"
-	"time"
-
-	"github.com/coreos/pkg/capnslog"
-	"github.com/julienschmidt/httprouter"
-	"github.com/prometheus/client_golang/prometheus"
-
-	"github.com/coreos/clair/config"
-	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/utils"
-)
-
-var (
-	log = capnslog.NewPackageLogger("github.com/coreos/clair", "api")
-
-	promResponseDurationMilliseconds = prometheus.NewHistogramVec(prometheus.HistogramOpts{
-		Name:    "clair_api_response_duration_milliseconds",
-		Help:    "The duration of time it takes to receieve and write a response to an API request",
-		Buckets: prometheus.ExponentialBuckets(9.375, 2, 10),
-	}, []string{"route", "code"})
-)
-
-func init() {
-	prometheus.MustRegister(promResponseDurationMilliseconds)
-}
-
-type Handler func(http.ResponseWriter, *http.Request, httprouter.Params, *RouteContext) (route string, status int)
-
-func HTTPHandler(handler Handler, ctx *RouteContext) httprouter.Handle {
-	return func(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
-		start := time.Now()
-		route, status := handler(w, r, p, ctx)
-		statusStr := strconv.Itoa(status)
-		if status == 0 {
-			statusStr = "???"
-		}
-		utils.PrometheusObserveTimeMilliseconds(promResponseDurationMilliseconds.WithLabelValues(route, statusStr), start)
-
-		log.Infof("%s \"%s %s\" %s (%s)", r.RemoteAddr, r.Method, r.RequestURI, statusStr, time.Since(start))
-	}
-}
-
-type RouteContext struct {
-	Store  database.Datastore
-	Config *config.APIConfig
-}

api/router.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -20,8 +20,8 @@ import (
 
 	"github.com/julienschmidt/httprouter"
 
-	"github.com/coreos/clair/api/context"
 	"github.com/coreos/clair/api/v1"
+	"github.com/coreos/clair/database"
 )
 
 // router is an HTTP router that forwards requests to the appropriate sub-router
@@ -31,9 +31,9 @@ type router map[string]*httprouter.Router
 // Let's hope we never have more than 99 API versions.
 const apiVersionLength = len("v99")
 
-func newAPIHandler(ctx *context.RouteContext) http.Handler {
+func newAPIHandler(cfg *Config, store database.Datastore) http.Handler {
 	router := make(router)
-	router["/v1"] = v1.NewRouter(ctx)
+	router["/v1"] = v1.NewRouter(store, cfg.PaginationKey)
 	return router
 }
 
@@ -56,21 +56,22 @@ func (rtr router) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	http.NotFound(w, r)
 }
 
-func newHealthHandler(ctx *context.RouteContext) http.Handler {
+func newHealthHandler(store database.Datastore) http.Handler {
 	router := httprouter.New()
-	router.GET("/health", context.HTTPHandler(getHealth, ctx))
+	router.GET("/health", healthHandler(store))
 	return router
 }
 
-func getHealth(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func healthHandler(store database.Datastore) httprouter.Handle {
-	header := w.Header()
+	return func(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
-	header.Set("Server", "clair")
+		header := w.Header()
+		header.Set("Server", "clair")
 
-	status := http.StatusInternalServerError
+		status := http.StatusInternalServerError
-	if ctx.Store.Ping() {
+		if store.Ping() {
-		status = http.StatusOK
+			status = http.StatusOK
+		}
+
+		w.WriteHeader(status)
 	}
-
-	w.WriteHeader(status)
-	return "health", status
 }

api/v1/models.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -26,7 +26,6 @@ import (
 
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt"
-	"github.com/coreos/clair/utils/types"
 )
 
 var log = capnslog.NewPackageLogger("github.com/coreos/clair", "v1")
@@ -109,9 +108,9 @@ type Vulnerability struct {
 }
 
 func (v Vulnerability) DatabaseModel() (database.Vulnerability, error) {
-	severity := types.Priority(v.Severity)
+	severity, err := database.NewSeverity(v.Severity)
-	if !severity.IsValid() {
+	if err != nil {
-		return database.Vulnerability{}, errors.New("Invalid severity")
+		return database.Vulnerability{}, err
 	}
 
 	var dbFeatures []database.FeatureVersion

api/v1/router.go

@@ -16,41 +16,83 @@
 package v1
 
 import (
-	"github.com/julienschmidt/httprouter"
+	"net/http"
+	"strconv"
+	"time"
 
-	"github.com/coreos/clair/api/context"
+	"github.com/julienschmidt/httprouter"
+	"github.com/prometheus/client_golang/prometheus"
+
+	"github.com/coreos/clair/database"
 )
 
+var (
+	promResponseDurationMilliseconds = prometheus.NewHistogramVec(prometheus.HistogramOpts{
+		Name:    "clair_api_response_duration_milliseconds",
+		Help:    "The duration of time it takes to receieve and write a response to an API request",
+		Buckets: prometheus.ExponentialBuckets(9.375, 2, 10),
+	}, []string{"route", "code"})
+)
+
+func init() {
+	prometheus.MustRegister(promResponseDurationMilliseconds)
+}
+
+type handler func(http.ResponseWriter, *http.Request, httprouter.Params, *context) (route string, status int)
+
+func httpHandler(h handler, ctx *context) httprouter.Handle {
+	return func(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
+		start := time.Now()
+		route, status := h(w, r, p, ctx)
+		statusStr := strconv.Itoa(status)
+		if status == 0 {
+			statusStr = "???"
+		}
+
+		promResponseDurationMilliseconds.
+			WithLabelValues(route, statusStr).
+			Observe(float64(time.Since(start).Nanoseconds()) / float64(time.Millisecond))
+
+		log.Infof("%s \"%s %s\" %s (%s)", r.RemoteAddr, r.Method, r.RequestURI, statusStr, time.Since(start))
+	}
+}
+
+type context struct {
+	Store         database.Datastore
+	PaginationKey string
+}
+
 // NewRouter creates an HTTP router for version 1 of the Clair API.
-func NewRouter(ctx *context.RouteContext) *httprouter.Router {
+func NewRouter(store database.Datastore, paginationKey string) *httprouter.Router {
 	router := httprouter.New()
+	ctx := &context{store, paginationKey}
 
 	// Layers
-	router.POST("/layers", context.HTTPHandler(postLayer, ctx))
+	router.POST("/layers", httpHandler(postLayer, ctx))
-	router.GET("/layers/:layerName", context.HTTPHandler(getLayer, ctx))
+	router.GET("/layers/:layerName", httpHandler(getLayer, ctx))
-	router.DELETE("/layers/:layerName", context.HTTPHandler(deleteLayer, ctx))
+	router.DELETE("/layers/:layerName", httpHandler(deleteLayer, ctx))
 
 	// Namespaces
-	router.GET("/namespaces", context.HTTPHandler(getNamespaces, ctx))
+	router.GET("/namespaces", httpHandler(getNamespaces, ctx))
 
 	// Vulnerabilities
-	router.GET("/namespaces/:namespaceName/vulnerabilities", context.HTTPHandler(getVulnerabilities, ctx))
+	router.GET("/namespaces/:namespaceName/vulnerabilities", httpHandler(getVulnerabilities, ctx))
-	router.POST("/namespaces/:namespaceName/vulnerabilities", context.HTTPHandler(postVulnerability, ctx))
+	router.POST("/namespaces/:namespaceName/vulnerabilities", httpHandler(postVulnerability, ctx))
-	router.GET("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName", context.HTTPHandler(getVulnerability, ctx))
+	router.GET("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName", httpHandler(getVulnerability, ctx))
-	router.PUT("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName", context.HTTPHandler(putVulnerability, ctx))
+	router.PUT("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName", httpHandler(putVulnerability, ctx))
-	router.DELETE("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName", context.HTTPHandler(deleteVulnerability, ctx))
+	router.DELETE("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName", httpHandler(deleteVulnerability, ctx))
 
 	// Fixes
-	router.GET("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName/fixes", context.HTTPHandler(getFixes, ctx))
+	router.GET("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName/fixes", httpHandler(getFixes, ctx))
-	router.PUT("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName/fixes/:fixName", context.HTTPHandler(putFix, ctx))
+	router.PUT("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName/fixes/:fixName", httpHandler(putFix, ctx))
-	router.DELETE("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName/fixes/:fixName", context.HTTPHandler(deleteFix, ctx))
+	router.DELETE("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName/fixes/:fixName", httpHandler(deleteFix, ctx))
 
 	// Notifications
-	router.GET("/notifications/:notificationName", context.HTTPHandler(getNotification, ctx))
+	router.GET("/notifications/:notificationName", httpHandler(getNotification, ctx))
-	router.DELETE("/notifications/:notificationName", context.HTTPHandler(deleteNotification, ctx))
+	router.DELETE("/notifications/:notificationName", httpHandler(deleteNotification, ctx))
 
 	// Metrics
-	router.GET("/metrics", context.HTTPHandler(getMetrics, ctx))
+	router.GET("/metrics", httpHandler(getMetrics, ctx))
 
 	return router
 }

api/v1/routes.go

@@ -25,11 +25,10 @@ import (
 	"github.com/julienschmidt/httprouter"
 	"github.com/prometheus/client_golang/prometheus"
 
-	"github.com/coreos/clair/api/context"
+	"github.com/coreos/clair"
 	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/utils"
+	"github.com/coreos/clair/pkg/commonerr"
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/tarutil"
-	"github.com/coreos/clair/worker"
 )
 
 const (
@@ -96,7 +95,7 @@ func writeResponse(w http.ResponseWriter, r *http.Request, status int, resp inte
 	}
 }
 
-func postLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func postLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	request := LayerEnvelope{}
 	err := decodeJSON(r, &request)
 	if err != nil {
@@ -109,16 +108,16 @@ func postLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx
 		return postLayerRoute, http.StatusBadRequest
 	}
 
-	err = worker.Process(ctx.Store, request.Layer.Format, request.Layer.Name, request.Layer.ParentName, request.Layer.Path, request.Layer.Headers)
+	err = clair.ProcessLayer(ctx.Store, request.Layer.Format, request.Layer.Name, request.Layer.ParentName, request.Layer.Path, request.Layer.Headers)
 	if err != nil {
-		if err == utils.ErrCouldNotExtract ||
+		if err == tarutil.ErrCouldNotExtract ||
-			err == utils.ErrExtractedFileTooBig ||
+			err == tarutil.ErrExtractedFileTooBig ||
-			err == worker.ErrUnsupported {
+			err == clair.ErrUnsupported {
 			writeResponse(w, r, statusUnprocessableEntity, LayerEnvelope{Error: &Error{err.Error()}})
 			return postLayerRoute, statusUnprocessableEntity
 		}
 
-		if _, badreq := err.(*cerrors.ErrBadRequest); badreq {
+		if _, badreq := err.(*commonerr.ErrBadRequest); badreq {
 			writeResponse(w, r, http.StatusBadRequest, LayerEnvelope{Error: &Error{err.Error()}})
 			return postLayerRoute, http.StatusBadRequest
 		}
@@ -133,17 +132,17 @@ func postLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx
 		Path:             request.Layer.Path,
 		Headers:          request.Layer.Headers,
 		Format:           request.Layer.Format,
-		IndexedByVersion: worker.Version,
+		IndexedByVersion: clair.Version,
 	}})
 	return postLayerRoute, http.StatusCreated
 }
 
-func getLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func getLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	_, withFeatures := r.URL.Query()["features"]
 	_, withVulnerabilities := r.URL.Query()["vulnerabilities"]
 
 	dbLayer, err := ctx.Store.FindLayer(p.ByName("layerName"), withFeatures, withVulnerabilities)
-	if err == cerrors.ErrNotFound {
+	if err == commonerr.ErrNotFound {
 		writeResponse(w, r, http.StatusNotFound, LayerEnvelope{Error: &Error{err.Error()}})
 		return getLayerRoute, http.StatusNotFound
 	} else if err != nil {
@@ -157,9 +156,9 @@ func getLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *
 	return getLayerRoute, http.StatusOK
 }
 
-func deleteLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func deleteLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	err := ctx.Store.DeleteLayer(p.ByName("layerName"))
-	if err == cerrors.ErrNotFound {
+	if err == commonerr.ErrNotFound {
 		writeResponse(w, r, http.StatusNotFound, LayerEnvelope{Error: &Error{err.Error()}})
 		return deleteLayerRoute, http.StatusNotFound
 	} else if err != nil {
@@ -171,7 +170,7 @@ func deleteLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ct
 	return deleteLayerRoute, http.StatusOK
 }
 
-func getNamespaces(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func getNamespaces(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	dbNamespaces, err := ctx.Store.ListNamespaces()
 	if err != nil {
 		writeResponse(w, r, http.StatusInternalServerError, NamespaceEnvelope{Error: &Error{err.Error()}})
@@ -189,7 +188,7 @@ func getNamespaces(w http.ResponseWriter, r *http.Request, p httprouter.Params,
 	return getNamespacesRoute, http.StatusOK
 }
 
-func getVulnerabilities(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func getVulnerabilities(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	query := r.URL.Query()
 
 	limitStrs, limitExists := query["limit"]
@@ -209,7 +208,7 @@ func getVulnerabilities(w http.ResponseWriter, r *http.Request, p httprouter.Par
 	page := 0
 	pageStrs, pageExists := query["page"]
 	if pageExists {
-		err = tokenUnmarshal(pageStrs[0], ctx.Config.PaginationKey, &page)
+		err = tokenUnmarshal(pageStrs[0], ctx.PaginationKey, &page)
 		if err != nil {
 			writeResponse(w, r, http.StatusBadRequest, VulnerabilityEnvelope{Error: &Error{"invalid page format: " + err.Error()}})
 			return getNotificationRoute, http.StatusBadRequest
@@ -223,7 +222,7 @@ func getVulnerabilities(w http.ResponseWriter, r *http.Request, p httprouter.Par
 	}
 
 	dbVulns, nextPage, err := ctx.Store.ListVulnerabilities(namespace, limit, page)
-	if err == cerrors.ErrNotFound {
+	if err == commonerr.ErrNotFound {
 		writeResponse(w, r, http.StatusNotFound, VulnerabilityEnvelope{Error: &Error{err.Error()}})
 		return getVulnerabilityRoute, http.StatusNotFound
 	} else if err != nil {
@@ -239,7 +238,7 @@ func getVulnerabilities(w http.ResponseWriter, r *http.Request, p httprouter.Par
 
 	var nextPageStr string
 	if nextPage != -1 {
-		nextPageBytes, err := tokenMarshal(nextPage, ctx.Config.PaginationKey)
+		nextPageBytes, err := tokenMarshal(nextPage, ctx.PaginationKey)
 		if err != nil {
 			writeResponse(w, r, http.StatusBadRequest, VulnerabilityEnvelope{Error: &Error{"failed to marshal token: " + err.Error()}})
 			return getNotificationRoute, http.StatusBadRequest
@@ -251,7 +250,7 @@ func getVulnerabilities(w http.ResponseWriter, r *http.Request, p httprouter.Par
 	return getVulnerabilitiesRoute, http.StatusOK
 }
 
-func postVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func postVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	request := VulnerabilityEnvelope{}
 	err := decodeJSON(r, &request)
 	if err != nil {
@@ -273,7 +272,7 @@ func postVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Para
 	err = ctx.Store.InsertVulnerabilities([]database.Vulnerability{vuln}, true)
 	if err != nil {
 		switch err.(type) {
-		case *cerrors.ErrBadRequest:
+		case *commonerr.ErrBadRequest:
 			writeResponse(w, r, http.StatusBadRequest, VulnerabilityEnvelope{Error: &Error{err.Error()}})
 			return postVulnerabilityRoute, http.StatusBadRequest
 		default:
@@ -286,11 +285,11 @@ func postVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Para
 	return postVulnerabilityRoute, http.StatusCreated
 }
 
-func getVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func getVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	_, withFixedIn := r.URL.Query()["fixedIn"]
 
 	dbVuln, err := ctx.Store.FindVulnerability(p.ByName("namespaceName"), p.ByName("vulnerabilityName"))
-	if err == cerrors.ErrNotFound {
+	if err == commonerr.ErrNotFound {
 		writeResponse(w, r, http.StatusNotFound, VulnerabilityEnvelope{Error: &Error{err.Error()}})
 		return getVulnerabilityRoute, http.StatusNotFound
 	} else if err != nil {
@@ -304,7 +303,7 @@ func getVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Param
 	return getVulnerabilityRoute, http.StatusOK
 }
 
-func putVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func putVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	request := VulnerabilityEnvelope{}
 	err := decodeJSON(r, &request)
 	if err != nil {
@@ -334,7 +333,7 @@ func putVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Param
 	err = ctx.Store.InsertVulnerabilities([]database.Vulnerability{vuln}, true)
 	if err != nil {
 		switch err.(type) {
-		case *cerrors.ErrBadRequest:
+		case *commonerr.ErrBadRequest:
 			writeResponse(w, r, http.StatusBadRequest, VulnerabilityEnvelope{Error: &Error{err.Error()}})
 			return putVulnerabilityRoute, http.StatusBadRequest
 		default:
@@ -347,9 +346,9 @@ func putVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Param
 	return putVulnerabilityRoute, http.StatusOK
 }
 
-func deleteVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func deleteVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	err := ctx.Store.DeleteVulnerability(p.ByName("namespaceName"), p.ByName("vulnerabilityName"))
-	if err == cerrors.ErrNotFound {
+	if err == commonerr.ErrNotFound {
 		writeResponse(w, r, http.StatusNotFound, VulnerabilityEnvelope{Error: &Error{err.Error()}})
 		return deleteVulnerabilityRoute, http.StatusNotFound
 	} else if err != nil {
@@ -361,9 +360,9 @@ func deleteVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Pa
 	return deleteVulnerabilityRoute, http.StatusOK
 }
 
-func getFixes(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func getFixes(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	dbVuln, err := ctx.Store.FindVulnerability(p.ByName("namespaceName"), p.ByName("vulnerabilityName"))
-	if err == cerrors.ErrNotFound {
+	if err == commonerr.ErrNotFound {
 		writeResponse(w, r, http.StatusNotFound, FeatureEnvelope{Error: &Error{err.Error()}})
 		return getFixesRoute, http.StatusNotFound
 	} else if err != nil {
@@ -376,7 +375,7 @@ func getFixes(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *
 	return getFixesRoute, http.StatusOK
 }
 
-func putFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func putFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	request := FeatureEnvelope{}
 	err := decodeJSON(r, &request)
 	if err != nil {
@@ -403,11 +402,11 @@ func putFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *co
 	err = ctx.Store.InsertVulnerabilityFixes(p.ByName("vulnerabilityNamespace"), p.ByName("vulnerabilityName"), []database.FeatureVersion{dbFix})
 	if err != nil {
 		switch err.(type) {
-		case *cerrors.ErrBadRequest:
+		case *commonerr.ErrBadRequest:
 			writeResponse(w, r, http.StatusBadRequest, FeatureEnvelope{Error: &Error{err.Error()}})
 			return putFixRoute, http.StatusBadRequest
 		default:
-			if err == cerrors.ErrNotFound {
+			if err == commonerr.ErrNotFound {
 				writeResponse(w, r, http.StatusNotFound, FeatureEnvelope{Error: &Error{err.Error()}})
 				return putFixRoute, http.StatusNotFound
 			}
@@ -420,9 +419,9 @@ func putFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *co
 	return putFixRoute, http.StatusOK
 }
 
-func deleteFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func deleteFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	err := ctx.Store.DeleteVulnerabilityFix(p.ByName("vulnerabilityNamespace"), p.ByName("vulnerabilityName"), p.ByName("fixName"))
-	if err == cerrors.ErrNotFound {
+	if err == commonerr.ErrNotFound {
 		writeResponse(w, r, http.StatusNotFound, FeatureEnvelope{Error: &Error{err.Error()}})
 		return deleteFixRoute, http.StatusNotFound
 	} else if err != nil {
@@ -434,7 +433,7 @@ func deleteFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx
 	return deleteFixRoute, http.StatusOK
 }
 
-func getNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func getNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	query := r.URL.Query()
 
 	limitStrs, limitExists := query["limit"]
@@ -452,14 +451,14 @@ func getNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params
 	page := database.VulnerabilityNotificationFirstPage
 	pageStrs, pageExists := query["page"]
 	if pageExists {
-		err := tokenUnmarshal(pageStrs[0], ctx.Config.PaginationKey, &page)
+		err := tokenUnmarshal(pageStrs[0], ctx.PaginationKey, &page)
 		if err != nil {
 			writeResponse(w, r, http.StatusBadRequest, NotificationEnvelope{Error: &Error{"invalid page format: " + err.Error()}})
 			return getNotificationRoute, http.StatusBadRequest
 		}
 		pageToken = pageStrs[0]
 	} else {
-		pageTokenBytes, err := tokenMarshal(page, ctx.Config.PaginationKey)
+		pageTokenBytes, err := tokenMarshal(page, ctx.PaginationKey)
 		if err != nil {
 			writeResponse(w, r, http.StatusBadRequest, NotificationEnvelope{Error: &Error{"failed to marshal token: " + err.Error()}})
 			return getNotificationRoute, http.StatusBadRequest
@@ -468,7 +467,7 @@ func getNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params
 	}
 
 	dbNotification, nextPage, err := ctx.Store.GetNotification(p.ByName("notificationName"), limit, page)
-	if err == cerrors.ErrNotFound {
+	if err == commonerr.ErrNotFound {
 		writeResponse(w, r, http.StatusNotFound, NotificationEnvelope{Error: &Error{err.Error()}})
 		return deleteNotificationRoute, http.StatusNotFound
 	} else if err != nil {
@@ -476,15 +475,15 @@ func getNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params
 		return getNotificationRoute, http.StatusInternalServerError
 	}
 
-	notification := NotificationFromDatabaseModel(dbNotification, limit, pageToken, nextPage, ctx.Config.PaginationKey)
+	notification := NotificationFromDatabaseModel(dbNotification, limit, pageToken, nextPage, ctx.PaginationKey)
 
 	writeResponse(w, r, http.StatusOK, NotificationEnvelope{Notification: &notification})
 	return getNotificationRoute, http.StatusOK
 }
 
-func deleteNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func deleteNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	err := ctx.Store.DeleteNotification(p.ByName("notificationName"))
-	if err == cerrors.ErrNotFound {
+	if err == commonerr.ErrNotFound {
 		writeResponse(w, r, http.StatusNotFound, NotificationEnvelope{Error: &Error{err.Error()}})
 		return deleteNotificationRoute, http.StatusNotFound
 	} else if err != nil {
@@ -496,7 +495,7 @@ func deleteNotification(w http.ResponseWriter, r *http.Request, p httprouter.Par
 	return deleteNotificationRoute, http.StatusOK
 }
 
-func getMetrics(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
+func getMetrics(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
 	prometheus.Handler().ServeHTTP(w, r)
 	return getMetricsRoute, 0
 }

clair.go

@@ -1,75 +0,0 @@
-// Copyright 2015 clair authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-// Package clair implements the ability to boot Clair with your own imports
-// that can dynamically register additional functionality.
-package clair
-
-import (
-	"math/rand"
-	"os"
-	"os/signal"
-	"syscall"
-	"time"
-
-	"github.com/coreos/clair/api"
-	"github.com/coreos/clair/api/context"
-	"github.com/coreos/clair/config"
-	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/notifier"
-	"github.com/coreos/clair/updater"
-	"github.com/coreos/clair/utils"
-	"github.com/coreos/pkg/capnslog"
-)
-
-var log = capnslog.NewPackageLogger("github.com/coreos/clair", "main")
-
-// Boot starts Clair. By exporting this function, anyone can import their own
-// custom fetchers/updaters into their own package and then call clair.Boot.
-func Boot(config *config.Config) {
-	rand.Seed(time.Now().UnixNano())
-	st := utils.NewStopper()
-
-	// Open database
-	db, err := database.Open(config.Database)
-	if err != nil {
-		log.Fatal(err)
-	}
-	defer db.Close()
-
-	// Start notifier
-	st.Begin()
-	go notifier.Run(config.Notifier, db, st)
-
-	// Start API
-	st.Begin()
-	go api.Run(config.API, &context.RouteContext{db, config.API}, st)
-	st.Begin()
-	go api.RunHealth(config.API, &context.RouteContext{db, config.API}, st)
-
-	// Start updater
-	st.Begin()
-	go updater.Run(config.Updater, db, st)
-
-	// Wait for interruption and shutdown gracefully.
-	waitForSignals(syscall.SIGINT, syscall.SIGTERM)
-	log.Info("Received interruption, gracefully stopping ...")
-	st.Stop()
-}
-
-func waitForSignals(signals ...os.Signal) {
-	interrupts := make(chan os.Signal, 1)
-	signal.Notify(interrupts, signals...)
-	<-interrupts
-}

cmd/clair/config.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package config
+package main
 
 import (
 	"errors"
@@ -20,21 +20,19 @@ import (
 	"os"
 	"time"
 
-	"github.com/fernet/fernet-go"
 	"gopkg.in/yaml.v2"
+
+	"github.com/coreos/clair"
+	"github.com/coreos/clair/api"
+	"github.com/coreos/clair/database"
+	"github.com/coreos/clair/ext/notification"
+	"github.com/fernet/fernet-go"
 )
 
-// ErrDatasourceNotLoaded is returned when the datasource variable in the configuration file is not loaded properly
+// ErrDatasourceNotLoaded is returned when the datasource variable in the
+// configuration file is not loaded properly
 var ErrDatasourceNotLoaded = errors.New("could not load configuration: no database source specified")
 
-// RegistrableComponentConfig is a configuration block that can be used to
-// determine which registrable component should be initialized and pass
-// custom configuration to it.
-type RegistrableComponentConfig struct {
-	Type    string
-	Options map[string]interface{}
-}
-
 // File represents a YAML configuration file that namespaces all Clair
 // configuration under the top-level "clair" key.
 type File struct {
@@ -43,57 +41,37 @@ type File struct {
 
 // Config is the global configuration for an instance of Clair.
 type Config struct {
-	Database RegistrableComponentConfig
+	Database database.RegistrableComponentConfig
-	Updater  *UpdaterConfig
+	Updater  *clair.UpdaterConfig
-	Notifier *NotifierConfig
+	Notifier *notification.Config
-	API      *APIConfig
+	API      *api.Config
-}
-
-// UpdaterConfig is the configuration for the Updater service.
-type UpdaterConfig struct {
-	Interval time.Duration
-}
-
-// NotifierConfig is the configuration for the Notifier service and its registered notifiers.
-type NotifierConfig struct {
-	Attempts         int
-	RenotifyInterval time.Duration
-	Params           map[string]interface{} `yaml:",inline"`
-}
-
-// APIConfig is the configuration for the API service.
-type APIConfig struct {
-	Port                      int
-	HealthPort                int
-	Timeout                   time.Duration
-	PaginationKey             string
-	CertFile, KeyFile, CAFile string
 }
 
 // DefaultConfig is a configuration that can be used as a fallback value.
 func DefaultConfig() Config {
 	return Config{
-		Database: RegistrableComponentConfig{
+		Database: database.RegistrableComponentConfig{
 			Type: "pgsql",
 		},
-		Updater: &UpdaterConfig{
+		Updater: &clair.UpdaterConfig{
 			Interval: 1 * time.Hour,
 		},
-		API: &APIConfig{
+		API: &api.Config{
 			Port:       6060,
 			HealthPort: 6061,
 			Timeout:    900 * time.Second,
 		},
-		Notifier: &NotifierConfig{
+		Notifier: &notification.Config{
 			Attempts:         5,
 			RenotifyInterval: 2 * time.Hour,
 		},
 	}
 }
 
-// Load is a shortcut to open a file, read it, and generate a Config.
+// LoadConfig is a shortcut to open a file, read it, and generate a Config.
+//
 // It supports relative and absolute paths. Given "", it returns DefaultConfig.
-func Load(path string) (config *Config, err error) {
+func LoadConfig(path string) (config *Config, err error) {
 	var cfgFile File
 	cfgFile.Clair = DefaultConfig()
 	if path == "" {

cmd/clair/main.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -16,67 +16,51 @@ package main
 
 import (
 	"flag"
+	"math/rand"
 	"os"
+	"os/exec"
+	"os/signal"
 	"runtime/pprof"
 	"strings"
+	"syscall"
+	"time"
 
 	"github.com/coreos/pkg/capnslog"
 
 	"github.com/coreos/clair"
-	"github.com/coreos/clair/config"
+	"github.com/coreos/clair/api"
-
+	"github.com/coreos/clair/database"
-	// Register components
+	"github.com/coreos/clair/pkg/stopper"
-	_ "github.com/coreos/clair/notifier/notifiers"
-
-	_ "github.com/coreos/clair/updater/fetchers/alpine"
-	_ "github.com/coreos/clair/updater/fetchers/debian"
-	_ "github.com/coreos/clair/updater/fetchers/oracle"
-	_ "github.com/coreos/clair/updater/fetchers/rhel"
-	_ "github.com/coreos/clair/updater/fetchers/ubuntu"
-	_ "github.com/coreos/clair/updater/metadata_fetchers/nvd"
-
-	_ "github.com/coreos/clair/worker/detectors/data/aci"
-	_ "github.com/coreos/clair/worker/detectors/data/docker"
-
-	_ "github.com/coreos/clair/worker/detectors/feature/apk"
-	_ "github.com/coreos/clair/worker/detectors/feature/dpkg"
-	_ "github.com/coreos/clair/worker/detectors/feature/rpm"
-
-	_ "github.com/coreos/clair/worker/detectors/namespace/alpinerelease"
-	_ "github.com/coreos/clair/worker/detectors/namespace/aptsources"
-	_ "github.com/coreos/clair/worker/detectors/namespace/lsbrelease"
-	_ "github.com/coreos/clair/worker/detectors/namespace/osrelease"
-	_ "github.com/coreos/clair/worker/detectors/namespace/redhatrelease"
 
+	// Register database driver.
 	_ "github.com/coreos/clair/database/pgsql"
+
+	// Register extensions.
+	_ "github.com/coreos/clair/ext/featurefmt/apk"
+	_ "github.com/coreos/clair/ext/featurefmt/dpkg"
+	_ "github.com/coreos/clair/ext/featurefmt/rpm"
+	_ "github.com/coreos/clair/ext/featurens/alpinerelease"
+	_ "github.com/coreos/clair/ext/featurens/aptsources"
+	_ "github.com/coreos/clair/ext/featurens/lsbrelease"
+	_ "github.com/coreos/clair/ext/featurens/osrelease"
+	_ "github.com/coreos/clair/ext/featurens/redhatrelease"
+	_ "github.com/coreos/clair/ext/imagefmt/aci"
+	_ "github.com/coreos/clair/ext/imagefmt/docker"
+	_ "github.com/coreos/clair/ext/notification/webhook"
+	_ "github.com/coreos/clair/ext/vulnmdsrc/nvd"
+	_ "github.com/coreos/clair/ext/vulnsrc/alpine"
+	_ "github.com/coreos/clair/ext/vulnsrc/debian"
+	_ "github.com/coreos/clair/ext/vulnsrc/oracle"
+	_ "github.com/coreos/clair/ext/vulnsrc/rhel"
+	_ "github.com/coreos/clair/ext/vulnsrc/ubuntu"
 )
 
 var log = capnslog.NewPackageLogger("github.com/coreos/clair/cmd/clair", "main")
 
-func main() {
+func waitForSignals(signals ...os.Signal) {
-	// Parse command-line arguments
+	interrupts := make(chan os.Signal, 1)
-	flag.CommandLine = flag.NewFlagSet(os.Args[0], flag.ExitOnError)
+	signal.Notify(interrupts, signals...)
-	flagConfigPath := flag.String("config", "/etc/clair/config.yaml", "Load configuration from the specified file.")
+	<-interrupts
-	flagCPUProfilePath := flag.String("cpu-profile", "", "Write a CPU profile to the specified file before exiting.")
-	flagLogLevel := flag.String("log-level", "info", "Define the logging level.")
-	flag.Parse()
-	// Load configuration
-	config, err := config.Load(*flagConfigPath)
-	if err != nil {
-		log.Fatalf("failed to load configuration: %s", err)
-	}
-
-	// Initialize logging system
-	logLevel, err := capnslog.ParseLevel(strings.ToUpper(*flagLogLevel))
-	capnslog.SetGlobalLogLevel(logLevel)
-	capnslog.SetFormatter(capnslog.NewPrettyFormatter(os.Stdout, false))
-
-	// Enable CPU Profiling if specified
-	if *flagCPUProfilePath != "" {
-		defer stopCPUProfiling(startCPUProfiling(*flagCPUProfilePath))
-	}
-
-	clair.Boot(config)
 }
 
 func startCPUProfiling(path string) *os.File {
@@ -100,3 +84,70 @@ func stopCPUProfiling(f *os.File) {
 	f.Close()
 	log.Info("stopped CPU profiling")
 }
+
+// Boot starts Clair instance with the provided config.
+func Boot(config *Config) {
+	rand.Seed(time.Now().UnixNano())
+	st := stopper.NewStopper()
+
+	// Open database
+	db, err := database.Open(config.Database)
+	if err != nil {
+		log.Fatal(err)
+	}
+	defer db.Close()
+
+	// Start notifier
+	st.Begin()
+	go clair.RunNotifier(config.Notifier, db, st)
+
+	// Start API
+	st.Begin()
+	go api.Run(config.API, db, st)
+	st.Begin()
+	go api.RunHealth(config.API, db, st)
+
+	// Start updater
+	st.Begin()
+	go clair.RunUpdater(config.Updater, db, st)
+
+	// Wait for interruption and shutdown gracefully.
+	waitForSignals(syscall.SIGINT, syscall.SIGTERM)
+	log.Info("Received interruption, gracefully stopping ...")
+	st.Stop()
+}
+
+func main() {
+	// Parse command-line arguments
+	flag.CommandLine = flag.NewFlagSet(os.Args[0], flag.ExitOnError)
+	flagConfigPath := flag.String("config", "/etc/clair/config.yaml", "Load configuration from the specified file.")
+	flagCPUProfilePath := flag.String("cpu-profile", "", "Write a CPU profile to the specified file before exiting.")
+	flagLogLevel := flag.String("log-level", "info", "Define the logging level.")
+	flag.Parse()
+
+	// Check for dependencies.
+	for _, bin := range []string{"git", "bzr", "rpm", "xz"} {
+		_, err := exec.LookPath(bin)
+		if err != nil {
+			log.Fatalf("failed to find dependency: %s", bin)
+		}
+	}
+
+	// Load configuration
+	config, err := LoadConfig(*flagConfigPath)
+	if err != nil {
+		log.Fatalf("failed to load configuration: %s", err)
+	}
+
+	// Initialize logging system
+	logLevel, err := capnslog.ParseLevel(strings.ToUpper(*flagLogLevel))
+	capnslog.SetGlobalLogLevel(logLevel)
+	capnslog.SetFormatter(capnslog.NewPrettyFormatter(os.Stdout, false))
+
+	// Enable CPU Profiling if specified
+	if *flagCPUProfilePath != "" {
+		defer stopCPUProfiling(startCPUProfiling(*flagCPUProfilePath))
+	}
+
+	Boot(config)
+}

database/database.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,15 +12,14 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-// Package database defines the Clair's models and a common interface for database implementations.
+// Package database defines the Clair's models and a common interface for
+// database implementations.
 package database
 
 import (
 	"errors"
 	"fmt"
 	"time"
-
-	"github.com/coreos/clair/config"
 )
 
 var (
@@ -29,15 +28,24 @@ var (
 	ErrBackendException = errors.New("database: an error occured when querying the backend")
 
 	// ErrInconsistent is an error that occurs when a database consistency check
-	// fails (ie. when an entity which is supposed to be unique is detected twice)
+	// fails (i.e. when an entity which is supposed to be unique is detected
+	// twice)
 	ErrInconsistent = errors.New("database: inconsistent database")
 )
 
+// RegistrableComponentConfig is a configuration block that can be used to
+// determine which registrable component should be initialized and pass custom
+// configuration to it.
+type RegistrableComponentConfig struct {
+	Type    string
+	Options map[string]interface{}
+}
+
 var drivers = make(map[string]Driver)
 
 // Driver is a function that opens a Datastore specified by its database driver type and specific
 // configuration.
-type Driver func(config.RegistrableComponentConfig) (Datastore, error)
+type Driver func(RegistrableComponentConfig) (Datastore, error)
 
 // Register makes a Constructor available by the provided name.
 //
@@ -54,7 +62,7 @@ func Register(name string, driver Driver) {
 }
 
 // Open opens a Datastore specified by a configuration.
-func Open(cfg config.RegistrableComponentConfig) (Datastore, error) {
+func Open(cfg RegistrableComponentConfig) (Datastore, error) {
 	driver, ok := drivers[cfg.Type]
 	if !ok {
 		return nil, fmt.Errorf("database: unknown Driver %q (forgotten configuration or import?)", cfg.Type)
@@ -62,123 +70,152 @@ func Open(cfg config.RegistrableComponentConfig) (Datastore, error) {
 	return driver(cfg)
 }
 
-// Datastore is the interface that describes a database backend implementation.
+// Datastore represents the required operations on a persistent data store for
+// a Clair deployment.
 type Datastore interface {
-	// # Namespace
 	// ListNamespaces returns the entire list of known Namespaces.
 	ListNamespaces() ([]Namespace, error)
 
-	// # Layer
 	// InsertLayer stores a Layer in the database.
-	// A Layer is uniquely identified by its Name. The Name and EngineVersion fields are mandatory.
+	//
-	// If a Parent is specified, it is expected that it has been retrieved using FindLayer.
+	// A Layer is uniquely identified by its Name.
-	// If a Layer that already exists is inserted and the EngineVersion of the given Layer is higher
+	// The Name and EngineVersion fields are mandatory.
-	// than the stored one, the stored Layer should be updated.
+	// If a Parent is specified, it is expected that it has been retrieved using
-	// The function has to be idempotent, inserting a layer that already exists shouln'd return an
+	// FindLayer.
-	// error.
+	// If a Layer that already exists is inserted and the EngineVersion of the
+	// given Layer is higher than the stored one, the stored Layer should be
+	// updated.
+	// The function has to be idempotent, inserting a layer that already exists
+	// shouldn't return an error.
 	InsertLayer(Layer) error
 
 	// FindLayer retrieves a Layer from the database.
-	// withFeatures specifies whether the Features field should be filled. When withVulnerabilities is
+	//
-	// true, the Features field should be filled and their AffectedBy fields should contain every
+	// When `withFeatures` is true, the Features field should be filled.
-	// vulnerabilities that affect them.
+	// When `withVulnerabilities` is true, the Features field should be filled
+	// and their AffectedBy fields should contain every vulnerabilities that
+	// affect them.
 	FindLayer(name string, withFeatures, withVulnerabilities bool) (Layer, error)
 
-	// DeleteLayer deletes a Layer from the database and every layers that are based on it,
+	// DeleteLayer deletes a Layer from the database and every layers that are
-	// recursively.
+	// based on it, recursively.
 	DeleteLayer(name string) error
 
-	// # Vulnerability
+	// ListVulnerabilities returns the list of vulnerabilities of a particular
-	// ListVulnerabilities returns the list of vulnerabilies of a certain Namespace.
+	// Namespace.
+	//
 	// The Limit and page parameters are used to paginate the return list.
-	// The first given page should be 0. The function will then return the next available page.
+	// The first given page should be 0.
-	// If there is no more page, -1 has to be returned.
+	// The function should return the next available page. If there are no more
+	// pages, -1 has to be returned.
 	ListVulnerabilities(namespaceName string, limit int, page int) ([]Vulnerability, int, error)
 
-	// InsertVulnerabilities stores the given Vulnerabilities in the database, updating them if
+	// InsertVulnerabilities stores the given Vulnerabilities in the database,
-	// necessary. A vulnerability is uniquely identified by its Namespace and its Name.
+	// updating them if necessary.
-	// The FixedIn field may only contain a partial list of Features that are affected by the
+	//
-	// Vulnerability, along with the version in which the vulnerability is fixed. It is the
+	// A vulnerability is uniquely identified by its Namespace and its Name.
-	// responsibility of the implementation to update the list properly. A version equals to
+	// The FixedIn field may only contain a partial list of Features that are
-	// types.MinVersion means that the given Feature is not being affected by the Vulnerability at
+	// affected by the Vulnerability, along with the version in which the
-	// all and thus, should be removed from the list. It is important that Features should be unique
+	// vulnerability is fixed. It is the responsibility of the implementation to
-	// in the FixedIn list. For example, it doesn't make sense to have two `openssl` Feature listed as
+	// update the list properly.
-	// a Vulnerability can only be fixed in one Version. This is true because Vulnerabilities and
+	// A version equals to versionfmt.MinVersion means that the given Feature is
-	// Features are Namespaced (i.e. specific to one operating system).
+	// not being affected by the Vulnerability at all and thus, should be removed
-	// Each vulnerability insertion or update has to create a Notification that will contain the
+	// from the list.
-	// old and the updated Vulnerability, unless createNotification equals to true.
+	// It is important that Features should be unique in the FixedIn list. For
+	// example, it doesn't make sense to have two `openssl` Feature listed as a
+	// Vulnerability can only be fixed in one Version. This is true because
+	// Vulnerabilities and Features are namespaced (i.e. specific to one
+	// operating system).
+	// Each vulnerability insertion or update has to create a Notification that
+	// will contain the old and the updated Vulnerability, unless
+	// createNotification equals to true.
 	InsertVulnerabilities(vulnerabilities []Vulnerability, createNotification bool) error
 
-	// FindVulnerability retrieves a Vulnerability from the database, including the FixedIn list.
+	// FindVulnerability retrieves a Vulnerability from the database, including
+	// the FixedIn list.
 	FindVulnerability(namespaceName, name string) (Vulnerability, error)
 
 	// DeleteVulnerability removes a Vulnerability from the database.
+	//
 	// It has to create a Notification that will contain the old Vulnerability.
 	DeleteVulnerability(namespaceName, name string) error
 
-	// InsertVulnerabilityFixes adds new FixedIn Feature or update the Versions of existing ones to
+	// InsertVulnerabilityFixes adds new FixedIn Feature or update the Versions
-	// the specified Vulnerability in the database.
+	// of existing ones to the specified Vulnerability in the database.
-	// It has has to create a Notification that will contain the old and the updated Vulnerability.
+	//
+	// It has has to create a Notification that will contain the old and the
+	// updated Vulnerability.
 	InsertVulnerabilityFixes(vulnerabilityNamespace, vulnerabilityName string, fixes []FeatureVersion) error
 
-	// DeleteVulnerabilityFix removes a FixedIn Feature from the specified Vulnerability in the
+	// DeleteVulnerabilityFix removes a FixedIn Feature from the specified
-	// database. It can be used to store the fact that a Vulnerability no longer affects the given
+	// Vulnerability in the database. It can be used to store the fact that a
-	// Feature in any Version.
+	// Vulnerability no longer affects the given Feature in any Version.
-	// It has has to create a Notification that will contain the old and the updated Vulnerability.
+	//
+	// It has has to create a Notification that will contain the old and the
+	// updated Vulnerability.
 	DeleteVulnerabilityFix(vulnerabilityNamespace, vulnerabilityName, featureName string) error
 
-	// # Notification
+	// GetAvailableNotification returns the Name, Created, Notified and Deleted
-	// GetAvailableNotification returns the Name, Created, Notified and Deleted fields of a
+	// fields of a Notification that should be handled.
-	// Notification that should be handled. The renotify interval defines how much time after being
+	//
-	// marked as Notified by SetNotificationNotified, a Notification that hasn't been deleted should
+	// The renotify interval defines how much time after being marked as Notified
-	// be returned again by this function. A Notification for which there is a valid Lock with the
+	// by SetNotificationNotified, a Notification that hasn't been deleted should
-	// same Name should not be returned.
+	// be returned again by this function.
+	// A Notification for which there is a valid Lock with the same Name should
+	// not be returned.
 	GetAvailableNotification(renotifyInterval time.Duration) (VulnerabilityNotification, error)
 
-	// GetNotification returns a Notification, including its OldVulnerability and NewVulnerability
+	// GetNotification returns a Notification, including its OldVulnerability and
-	// fields. On these Vulnerabilities, LayersIntroducingVulnerability should be filled with
+	// NewVulnerability fields.
-	// every Layer that introduces the Vulnerability (i.e. adds at least one affected FeatureVersion).
+	//
-	// The Limit and page parameters are used to paginate LayersIntroducingVulnerability. The first
+	// On these Vulnerabilities, LayersIntroducingVulnerability should be filled
-	// given page should be VulnerabilityNotificationFirstPage. The function will then return the next
+	// with every Layer that introduces the Vulnerability (i.e. adds at least one
-	// availage page. If there is no more page, NoVulnerabilityNotificationPage has to be returned.
+	// affected FeatureVersion).
+	// The Limit and page parameters are used to paginate
+	// LayersIntroducingVulnerability. The first given page should be
+	// VulnerabilityNotificationFirstPage. The function will then return the next
+	// available page. If there is no more page, NoVulnerabilityNotificationPage
+	// has to be returned.
 	GetNotification(name string, limit int, page VulnerabilityNotificationPageNumber) (VulnerabilityNotification, VulnerabilityNotificationPageNumber, error)
 
-	// SetNotificationNotified marks a Notification as notified and thus, makes it unavailable for
+	// SetNotificationNotified marks a Notification as notified and thus, makes
-	// GetAvailableNotification, until the renotify duration is elapsed.
+	// it unavailable for GetAvailableNotification, until the renotify duration
+	// is elapsed.
 	SetNotificationNotified(name string) error
 
-	// DeleteNotification marks a Notification as deleted, and thus, makes it unavailable for
+	// DeleteNotification marks a Notification as deleted, and thus, makes it
-	// GetAvailableNotification.
+	// unavailable for GetAvailableNotification.
 	DeleteNotification(name string) error
 
-	// # Key/Value
 	// InsertKeyValue stores or updates a simple key/value pair in the database.
 	InsertKeyValue(key, value string) error
 
 	// GetKeyValue retrieves a value from the database from the given key.
+	//
 	// It returns an empty string if there is no such key.
 	GetKeyValue(key string) (string, error)
 
-	// # Lock
+	// Lock creates or renew a Lock in the database with the given name, owner
-	// Lock creates or renew a Lock in the database with the given name, owner and duration.
+	// and duration.
-	// After the specified duration, the Lock expires by itself if it hasn't been unlocked, and thus,
+	//
-	// let other users create a Lock with the same name. However, the owner can renew its Lock by
+	// After the specified duration, the Lock expires by itself if it hasn't been
-	// setting renew to true. Lock should not block, it should instead returns whether the Lock has
+	// unlocked, and thus, let other users create a Lock with the same name.
-	// been successfully acquired/renewed. If it's the case, the expiration time of that Lock is
+	// However, the owner can renew its Lock by setting renew to true.
-	// returned as well.
+	// Lock should not block, it should instead returns whether the Lock has been
+	// successfully acquired/renewed. If it's the case, the expiration time of
+	// that Lock is returned as well.
 	Lock(name string, owner string, duration time.Duration, renew bool) (bool, time.Time)
 
 	// Unlock releases an existing Lock.
 	Unlock(name, owner string)
 
-	// FindLock returns the owner of a Lock specified by the name, and its experation time if it
+	// FindLock returns the owner of a Lock specified by the name, and its
-	// exists.
+	// expiration time if it exists.
 	FindLock(name string) (string, time.Time, error)
 
-	// # Miscellaneous
 	// Ping returns the health status of the database.
 	Ping() bool
 
-	// Close closes the database and free any allocated resource.
+	// Close closes the database and frees any allocated resource.
 	Close()
 }

database/models.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -18,8 +18,6 @@ import (
 	"database/sql/driver"
 	"encoding/json"
 	"time"
-
-	"github.com/coreos/clair/utils/types"
 )
 
 // ID is only meant to be used by database implementations and should never be used for anything else.
@@ -70,7 +68,7 @@ type Vulnerability struct {
 
 	Description string
 	Link        string
-	Severity    types.Priority
+	Severity    Severity
 
 	Metadata MetadataMap
 

database/pgsql/complex_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -28,8 +28,6 @@ import (
 
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
-	"github.com/coreos/clair/utils"
-	"github.com/coreos/clair/utils/types"
 )
 
 const (
@@ -93,7 +91,7 @@ func TestRaceAffects(t *testing.T) {
 					Version: strconv.Itoa(version),
 				},
 			},
-			Severity: types.Unknown,
+			Severity: database.UnknownSeverity,
 		}
 
 		vulnerabilities[version] = append(vulnerabilities[version], vulnerability)
@@ -157,7 +155,7 @@ func TestRaceAffects(t *testing.T) {
 			}
 		}
 
-		assert.Len(t, utils.CompareStringLists(expectedAffectedNames, actualAffectedNames), 0)
+		assert.Len(t, compareStringLists(expectedAffectedNames, actualAffectedNames), 0)
-		assert.Len(t, utils.CompareStringLists(actualAffectedNames, expectedAffectedNames), 0)
+		assert.Len(t, compareStringLists(actualAffectedNames, expectedAffectedNames), 0)
 	}
 }

database/pgsql/feature.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -21,12 +21,12 @@ import (
 
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt"
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/commonerr"
 )
 
 func (pgSQL *pgSQL) insertFeature(feature database.Feature) (int, error) {
 	if feature.Name == "" {
-		return 0, cerrors.NewBadRequestError("could not find/insert invalid Feature")
+		return 0, commonerr.NewBadRequestError("could not find/insert invalid Feature")
 	}
 
 	// Do cache lookup.
@@ -65,7 +65,7 @@ func (pgSQL *pgSQL) insertFeature(feature database.Feature) (int, error) {
 func (pgSQL *pgSQL) insertFeatureVersion(fv database.FeatureVersion) (id int, err error) {
 	err = versionfmt.Valid(fv.Feature.Namespace.VersionFormat, fv.Version)
 	if err != nil {
-		return 0, cerrors.NewBadRequestError("could not find/insert invalid FeatureVersion")
+		return 0, commonerr.NewBadRequestError("could not find/insert invalid FeatureVersion")
 	}
 
 	// Do cache lookup.

database/pgsql/keyvalue.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -18,14 +18,14 @@ import (
 	"database/sql"
 	"time"
 
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/commonerr"
 )
 
 // InsertKeyValue stores (or updates) a single key / value tuple.
 func (pgSQL *pgSQL) InsertKeyValue(key, value string) (err error) {
 	if key == "" || value == "" {
 		log.Warning("could not insert a flag which has an empty name or value")
-		return cerrors.NewBadRequestError("could not insert a flag which has an empty name or value")
+		return commonerr.NewBadRequestError("could not insert a flag which has an empty name or value")
 	}
 
 	defer observeQueryTime("InsertKeyValue", "all", time.Now())

database/pgsql/layer.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -22,8 +22,7 @@ import (
 	"github.com/guregu/null/zero"
 
 	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/utils"
+	"github.com/coreos/clair/pkg/commonerr"
-	cerrors "github.com/coreos/clair/utils/errors"
 )
 
 func (pgSQL *pgSQL) FindLayer(name string, withFeatures, withVulnerabilities bool) (database.Layer, error) {
@@ -247,12 +246,12 @@ func (pgSQL *pgSQL) InsertLayer(layer database.Layer) error {
 	// Verify parameters
 	if layer.Name == "" {
 		log.Warning("could not insert a layer which has an empty Name")
-		return cerrors.NewBadRequestError("could not insert a layer which has an empty Name")
+		return commonerr.NewBadRequestError("could not insert a layer which has an empty Name")
 	}
 
 	// Get a potentially existing layer.
 	existingLayer, err := pgSQL.FindLayer(layer.Name, true, false)
-	if err != nil && err != cerrors.ErrNotFound {
+	if err != nil && err != commonerr.ErrNotFound {
 		return err
 	} else if err == nil {
 		if existingLayer.EngineVersion >= layer.EngineVersion {
@@ -271,7 +270,7 @@ func (pgSQL *pgSQL) InsertLayer(layer database.Layer) error {
 	if layer.Parent != nil {
 		if layer.Parent.ID == 0 {
 			log.Warning("Parent is expected to be retrieved from database when inserting a layer.")
-			return cerrors.NewBadRequestError("Parent is expected to be retrieved from database when inserting a layer.")
+			return commonerr.NewBadRequestError("Parent is expected to be retrieved from database when inserting a layer.")
 		}
 
 		parentID = zero.IntFrom(int64(layer.Parent.ID))
@@ -362,8 +361,8 @@ func (pgSQL *pgSQL) updateDiffFeatureVersions(tx *sql.Tx, layer, existingLayer *
 		parentLayerFeaturesMapNV, parentLayerFeaturesNV := createNV(layer.Parent.Features)
 
 		// Calculate the added and deleted FeatureVersions name:version.
-		addNV := utils.CompareStringLists(layerFeaturesNV, parentLayerFeaturesNV)
+		addNV := compareStringLists(layerFeaturesNV, parentLayerFeaturesNV)
-		delNV := utils.CompareStringLists(parentLayerFeaturesNV, layerFeaturesNV)
+		delNV := compareStringLists(parentLayerFeaturesNV, layerFeaturesNV)
 
 		// Fill the structures containing the added and deleted FeatureVersions.
 		for _, nv := range addNV {
@@ -429,7 +428,7 @@ func (pgSQL *pgSQL) DeleteLayer(name string) error {
 	}
 
 	if affected <= 0 {
-		return cerrors.ErrNotFound
+		return commonerr.ErrNotFound
 	}
 
 	return nil

database/pgsql/layer_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -22,8 +22,7 @@ import (
 
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/commonerr"
-	"github.com/coreos/clair/utils/types"
 )
 
 func TestFindLayer(t *testing.T) {
@@ -91,7 +90,7 @@ func TestFindLayer(t *testing.T) {
 				if assert.Len(t, featureVersion.AffectedBy, 1) {
 					assert.Equal(t, "debian:7", featureVersion.AffectedBy[0].Namespace.Name)
 					assert.Equal(t, "CVE-OPENSSL-1-DEB7", featureVersion.AffectedBy[0].Name)
-					assert.Equal(t, types.High, featureVersion.AffectedBy[0].Severity)
+					assert.Equal(t, database.HighSeverity, featureVersion.AffectedBy[0].Severity)
 					assert.Equal(t, "A vulnerability affecting OpenSSL < 2.0 on Debian 7.0", featureVersion.AffectedBy[0].Description)
 					assert.Equal(t, "http://google.com/#q=CVE-OPENSSL-1-DEB7", featureVersion.AffectedBy[0].Link)
 					assert.Equal(t, "2.0", featureVersion.AffectedBy[0].FixedBy)
@@ -363,19 +362,19 @@ func testInsertLayerUpdate(t *testing.T, datastore database.Datastore) {
 
 func testInsertLayerDelete(t *testing.T, datastore database.Datastore) {
 	err := datastore.DeleteLayer("TestInsertLayerX")
-	assert.Equal(t, cerrors.ErrNotFound, err)
+	assert.Equal(t, commonerr.ErrNotFound, err)
 
 	err = datastore.DeleteLayer("TestInsertLayer3")
 	assert.Nil(t, err)
 
 	_, err = datastore.FindLayer("TestInsertLayer3", false, false)
-	assert.Equal(t, cerrors.ErrNotFound, err)
+	assert.Equal(t, commonerr.ErrNotFound, err)
 
 	_, err = datastore.FindLayer("TestInsertLayer4a", false, false)
-	assert.Equal(t, cerrors.ErrNotFound, err)
+	assert.Equal(t, commonerr.ErrNotFound, err)
 
 	_, err = datastore.FindLayer("TestInsertLayer4b", true, false)
-	assert.Equal(t, cerrors.ErrNotFound, err)
+	assert.Equal(t, commonerr.ErrNotFound, err)
 }
 
 func cmpFV(a, b database.FeatureVersion) bool {

database/pgsql/lock.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -17,7 +17,7 @@ package pgsql
 import (
 	"time"
 
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/commonerr"
 )
 
 // Lock tries to set a temporary lock in the database.
@@ -80,7 +80,7 @@ func (pgSQL *pgSQL) Unlock(name, owner string) {
 func (pgSQL *pgSQL) FindLock(name string) (string, time.Time, error) {
 	if name == "" {
 		log.Warning("could not find an invalid lock")
-		return "", time.Time{}, cerrors.NewBadRequestError("could not find an invalid lock")
+		return "", time.Time{}, commonerr.NewBadRequestError("could not find an invalid lock")
 	}
 
 	defer observeQueryTime("FindLock", "all", time.Now())

database/pgsql/namespace.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -18,12 +18,12 @@ import (
 	"time"
 
 	"github.com/coreos/clair/database"
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/commonerr"
 )
 
 func (pgSQL *pgSQL) insertNamespace(namespace database.Namespace) (int, error) {
 	if namespace.Name == "" {
-		return 0, cerrors.NewBadRequestError("could not find/insert invalid Namespace")
+		return 0, commonerr.NewBadRequestError("could not find/insert invalid Namespace")
 	}
 
 	if pgSQL.cache != nil {

database/pgsql/notification.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -19,7 +19,7 @@ import (
 	"time"
 
 	"github.com/coreos/clair/database"
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/commonerr"
 	"github.com/guregu/null/zero"
 	"github.com/pborman/uuid"
 )
@@ -242,7 +242,7 @@ func (pgSQL *pgSQL) DeleteNotification(name string) error {
 	}
 
 	if affected <= 0 {
-		return cerrors.ErrNotFound
+		return commonerr.ErrNotFound
 	}
 
 	return nil

database/pgsql/notification_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -23,8 +23,7 @@ import (
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/commonerr"
-	"github.com/coreos/clair/utils/types"
 )
 
 func TestNotification(t *testing.T) {
@@ -37,7 +36,7 @@ func TestNotification(t *testing.T) {
 
 	// Try to get a notification when there is none.
 	_, err = datastore.GetAvailableNotification(time.Second)
-	assert.Equal(t, cerrors.ErrNotFound, err)
+	assert.Equal(t, commonerr.ErrNotFound, err)
 
 	// Create some data.
 	f1 := database.Feature{
@@ -126,7 +125,7 @@ func TestNotification(t *testing.T) {
 		// Verify the renotify behaviour.
 		if assert.Nil(t, datastore.SetNotificationNotified(notification.Name)) {
 			_, err := datastore.GetAvailableNotification(time.Second)
-			assert.Equal(t, cerrors.ErrNotFound, err)
+			assert.Equal(t, commonerr.ErrNotFound, err)
 
 			time.Sleep(50 * time.Millisecond)
 			notificationB, err := datastore.GetAvailableNotification(20 * time.Millisecond)
@@ -164,12 +163,12 @@ func TestNotification(t *testing.T) {
 		assert.Nil(t, datastore.DeleteNotification(notification.Name))
 
 		_, err = datastore.GetAvailableNotification(time.Millisecond)
-		assert.Equal(t, cerrors.ErrNotFound, err)
+		assert.Equal(t, commonerr.ErrNotFound, err)
 	}
 
 	// Update a vulnerability and ensure that the old/new vulnerabilities are correct.
 	v1b := v1
-	v1b.Severity = types.High
+	v1b.Severity = database.HighSeverity
 	v1b.FixedIn = []database.FeatureVersion{
 		{
 			Feature: f1,

database/pgsql/pgsql.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -31,11 +31,9 @@ import (
 	"github.com/prometheus/client_golang/prometheus"
 	"github.com/remind101/migrate"
 
-	"github.com/coreos/clair/config"
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/database/pgsql/migrations"
-	"github.com/coreos/clair/utils"
+	"github.com/coreos/clair/pkg/commonerr"
-	cerrors "github.com/coreos/clair/utils/errors"
 )
 
 var (
@@ -115,11 +113,13 @@ type Config struct {
 	FixturePath             string
 }
 
-// openDatabase opens a PostgresSQL-backed Datastore using the given configuration.
+// openDatabase opens a PostgresSQL-backed Datastore using the given
-// It immediately every necessary migrations. If ManageDatabaseLifecycle is specified,
+// configuration.
-// the database will be created first. If FixturePath is specified, every SQL queries that are
+//
-// present insides will be executed.
+// It immediately runs all necessary migrations. If ManageDatabaseLifecycle is
-func openDatabase(registrableComponentConfig config.RegistrableComponentConfig) (database.Datastore, error) {
+// specified, the database will be created first. If FixturePath is specified,
+// every SQL queries that are present insides will be executed.
+func openDatabase(registrableComponentConfig database.RegistrableComponentConfig) (database.Datastore, error) {
 	var pg pgSQL
 	var err error
 
@@ -196,12 +196,12 @@ func openDatabase(registrableComponentConfig config.RegistrableComponentConfig)
 
 func parseConnectionString(source string) (dbName string, pgSourceURL string, err error) {
 	if source == "" {
-		return "", "", cerrors.NewBadRequestError("pgsql: no database connection string specified")
+		return "", "", commonerr.NewBadRequestError("pgsql: no database connection string specified")
 	}
 
 	sourceURL, err := url.Parse(source)
 	if err != nil {
-		return "", "", cerrors.NewBadRequestError("pgsql: database connection string is not a valid URL")
+		return "", "", commonerr.NewBadRequestError("pgsql: database connection string is not a valid URL")
 	}
 
 	dbName = strings.TrimPrefix(sourceURL.Path, "/")
@@ -280,7 +280,7 @@ func handleError(desc string, err error) error {
 	}
 
 	if err == sql.ErrNoRows {
-		return cerrors.ErrNotFound
+		return commonerr.ErrNotFound
 	}
 
 	log.Errorf("%s: %v", desc, err)
@@ -300,5 +300,7 @@ func isErrUniqueViolation(err error) bool {
 }
 
 func observeQueryTime(query, subquery string, start time.Time) {
-	utils.PrometheusObserveTimeMilliseconds(promQueryDurationMilliseconds.WithLabelValues(query, subquery), start)
+	promQueryDurationMilliseconds.
+		WithLabelValues(query, subquery).
+		Observe(float64(time.Since(start).Nanoseconds()) / float64(time.Millisecond))
 }

database/pgsql/pgsql_test.go

@@ -21,8 +21,9 @@ import (
 	"runtime"
 	"strings"
 
-	"github.com/coreos/clair/config"
 	"github.com/pborman/uuid"
+
+	"github.com/coreos/clair/database"
 )
 
 func openDatabaseForTest(testName string, loadFixture bool) (*pgSQL, error) {
@@ -34,7 +35,7 @@ func openDatabaseForTest(testName string, loadFixture bool) (*pgSQL, error) {
 	return datastore, nil
 }
 
-func generateTestConfig(testName string, loadFixture bool) config.RegistrableComponentConfig {
+func generateTestConfig(testName string, loadFixture bool) database.RegistrableComponentConfig {
 	dbName := "test_" + strings.ToLower(testName) + "_" + strings.Replace(uuid.New(), "-", "_", -1)
 
 	var fixturePath string
@@ -48,7 +49,7 @@ func generateTestConfig(testName string, loadFixture bool) config.RegistrableCom
 		source = fmt.Sprintf(sourceEnv, dbName)
 	}
 
-	return config.RegistrableComponentConfig{
+	return database.RegistrableComponentConfig{
 		Options: map[string]interface{}{
 			"source":                  source,
 			"cachesize":               0,

database/pgsql/vulnerability.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -17,17 +17,54 @@ package pgsql
 import (
 	"database/sql"
 	"encoding/json"
-	"fmt"
 	"reflect"
 	"time"
 
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt"
-	"github.com/coreos/clair/utils"
+	"github.com/coreos/clair/pkg/commonerr"
-	cerrors "github.com/coreos/clair/utils/errors"
 	"github.com/guregu/null/zero"
 )
 
+// compareStringLists returns the strings that are present in X but not in Y.
+func compareStringLists(X, Y []string) []string {
+	m := make(map[string]bool)
+
+	for _, y := range Y {
+		m[y] = true
+	}
+
+	diff := []string{}
+	for _, x := range X {
+		if m[x] {
+			continue
+		}
+
+		diff = append(diff, x)
+		m[x] = true
+	}
+
+	return diff
+}
+
+func compareStringListsInBoth(X, Y []string) []string {
+	m := make(map[string]struct{})
+
+	for _, y := range Y {
+		m[y] = struct{}{}
+	}
+
+	diff := []string{}
+	for _, x := range X {
+		if _, e := m[x]; e {
+			diff = append(diff, x)
+			delete(m, x)
+		}
+	}
+
+	return diff
+}
+
 func (pgSQL *pgSQL) ListVulnerabilities(namespaceName string, limit int, startID int) ([]database.Vulnerability, int, error) {
 	defer observeQueryTime("listVulnerabilities", "all", time.Now())
 
@@ -37,7 +74,7 @@ func (pgSQL *pgSQL) ListVulnerabilities(namespaceName string, limit int, startID
 	if err != nil {
 		return nil, -1, handleError("searchNamespace", err)
 	} else if id == 0 {
-		return nil, -1, cerrors.ErrNotFound
+		return nil, -1, commonerr.ErrNotFound
 	}
 
 	// Query.
@@ -130,7 +167,7 @@ func scanVulnerability(queryer Queryer, queryName string, vulnerabilityRow *sql.
 	}
 
 	if vulnerability.ID == 0 {
-		return vulnerability, cerrors.ErrNotFound
+		return vulnerability, commonerr.ErrNotFound
 	}
 
 	// Query the FixedIn FeatureVersion now.
@@ -195,13 +232,9 @@ func (pgSQL *pgSQL) insertVulnerability(vulnerability database.Vulnerability, on
 
 	// Verify parameters
 	if vulnerability.Name == "" || vulnerability.Namespace.Name == "" {
-		return cerrors.NewBadRequestError("insertVulnerability needs at least the Name and the Namespace")
+		return commonerr.NewBadRequestError("insertVulnerability needs at least the Name and the Namespace")
-	}
-	if !onlyFixedIn && !vulnerability.Severity.IsValid() {
-		msg := fmt.Sprintf("could not insert a vulnerability that has an invalid Severity: %s", vulnerability.Severity)
-		log.Warning(msg)
-		return cerrors.NewBadRequestError(msg)
 	}
+
 	for i := 0; i < len(vulnerability.FixedIn); i++ {
 		fifv := &vulnerability.FixedIn[i]
 
@@ -212,7 +245,7 @@ func (pgSQL *pgSQL) insertVulnerability(vulnerability database.Vulnerability, on
 		} else if fifv.Feature.Namespace.Name != vulnerability.Namespace.Name {
 			msg := "could not insert an invalid vulnerability that contains FixedIn FeatureVersion that are not in the same namespace as the Vulnerability"
 			log.Warning(msg)
-			return cerrors.NewBadRequestError(msg)
+			return commonerr.NewBadRequestError(msg)
 		}
 	}
 
@@ -228,7 +261,7 @@ func (pgSQL *pgSQL) insertVulnerability(vulnerability database.Vulnerability, on
 
 	// Find existing vulnerability and its Vulnerability_FixedIn_Features (for update).
 	existingVulnerability, err := findVulnerability(tx, vulnerability.Namespace.Name, vulnerability.Name, true)
-	if err != nil && err != cerrors.ErrNotFound {
+	if err != nil && err != commonerr.ErrNotFound {
 		tx.Rollback()
 		return err
 	}
@@ -237,7 +270,7 @@ func (pgSQL *pgSQL) insertVulnerability(vulnerability database.Vulnerability, on
 		// Because this call tries to update FixedIn FeatureVersion, import all other data from the
 		// existing one.
 		if existingVulnerability.ID == 0 {
-			return cerrors.ErrNotFound
+			return commonerr.ErrNotFound
 		}
 
 		fixedIn := vulnerability.FixedIn
@@ -271,8 +304,9 @@ func (pgSQL *pgSQL) insertVulnerability(vulnerability database.Vulnerability, on
 			return handleError("removeVulnerability", err)
 		}
 	} else {
-		// The vulnerability is new, we don't want to have any types.MinVersion as they are only used
+		// The vulnerability is new, we don't want to have any
-		// for diffing existing vulnerabilities.
+		// versionfmt.MinVersion as they are only used for diffing existing
+		// vulnerabilities.
 		var fixedIn []database.FeatureVersion
 		for _, fv := range vulnerability.FixedIn {
 			if fv.Version != versionfmt.MinVersion {
@@ -345,8 +379,8 @@ func applyFixedInDiff(currentList, diff []database.FeatureVersion) ([]database.F
 	currentMap, currentNames := createFeatureVersionNameMap(currentList)
 	diffMap, diffNames := createFeatureVersionNameMap(diff)
 
-	addedNames := utils.CompareStringLists(diffNames, currentNames)
+	addedNames := compareStringLists(diffNames, currentNames)
-	inBothNames := utils.CompareStringListsInBoth(diffNames, currentNames)
+	inBothNames := compareStringListsInBoth(diffNames, currentNames)
 
 	different := false
 

database/pgsql/vulnerability_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -23,8 +23,7 @@ import (
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/commonerr"
-	"github.com/coreos/clair/utils/types"
 )
 
 func TestFindVulnerability(t *testing.T) {
@@ -37,14 +36,14 @@ func TestFindVulnerability(t *testing.T) {
 
 	// Find a vulnerability that does not exist.
 	_, err = datastore.FindVulnerability("", "")
-	assert.Equal(t, cerrors.ErrNotFound, err)
+	assert.Equal(t, commonerr.ErrNotFound, err)
 
 	// Find a normal vulnerability.
 	v1 := database.Vulnerability{
 		Name:        "CVE-OPENSSL-1-DEB7",
 		Description: "A vulnerability affecting OpenSSL < 2.0 on Debian 7.0",
 		Link:        "http://google.com/#q=CVE-OPENSSL-1-DEB7",
-		Severity:    types.High,
+		Severity:    database.HighSeverity,
 		Namespace: database.Namespace{
 			Name:          "debian:7",
 			VersionFormat: dpkg.ParserName,
@@ -74,7 +73,7 @@ func TestFindVulnerability(t *testing.T) {
 			Name:          "debian:7",
 			VersionFormat: dpkg.ParserName,
 		},
-		Severity: types.Unknown,
+		Severity: database.UnknownSeverity,
 	}
 
 	v2f, err := datastore.FindVulnerability("debian:7", "CVE-NOPE")
@@ -93,15 +92,15 @@ func TestDeleteVulnerability(t *testing.T) {
 
 	// Delete non-existing Vulnerability.
 	err = datastore.DeleteVulnerability("TestDeleteVulnerabilityNamespace1", "CVE-OPENSSL-1-DEB7")
-	assert.Equal(t, cerrors.ErrNotFound, err)
+	assert.Equal(t, commonerr.ErrNotFound, err)
 	err = datastore.DeleteVulnerability("debian:7", "TestDeleteVulnerabilityVulnerability1")
-	assert.Equal(t, cerrors.ErrNotFound, err)
+	assert.Equal(t, commonerr.ErrNotFound, err)
 
 	// Delete Vulnerability.
 	err = datastore.DeleteVulnerability("debian:7", "CVE-OPENSSL-1-DEB7")
 	if assert.Nil(t, err) {
 		_, err := datastore.FindVulnerability("debian:7", "CVE-OPENSSL-1-DEB7")
-		assert.Equal(t, cerrors.ErrNotFound, err)
+		assert.Equal(t, commonerr.ErrNotFound, err)
 	}
 }
 
@@ -180,30 +179,24 @@ func TestInsertVulnerability(t *testing.T) {
 			Name:      "",
 			Namespace: n1,
 			FixedIn:   []database.FeatureVersion{f1},
-			Severity:  types.Unknown,
+			Severity:  database.UnknownSeverity,
 		},
 		{
 			Name:      "TestInsertVulnerability0",
 			Namespace: database.Namespace{},
 			FixedIn:   []database.FeatureVersion{f1},
-			Severity:  types.Unknown,
+			Severity:  database.UnknownSeverity,
 		},
 		{
 			Name:      "TestInsertVulnerability0-",
 			Namespace: database.Namespace{},
 			FixedIn:   []database.FeatureVersion{f1},
 		},
-		{
-			Name:      "TestInsertVulnerability0",
-			Namespace: n1,
-			FixedIn:   []database.FeatureVersion{f1},
-			Severity:  types.Priority(""),
-		},
 		{
 			Name:      "TestInsertVulnerability0",
 			Namespace: n1,
 			FixedIn:   []database.FeatureVersion{f2},
-			Severity:  types.Unknown,
+			Severity:  database.UnknownSeverity,
 		},
 	} {
 		err := datastore.InsertVulnerabilities([]database.Vulnerability{vulnerability}, true)
@@ -223,7 +216,7 @@ func TestInsertVulnerability(t *testing.T) {
 		Name:        "TestInsertVulnerability1",
 		Namespace:   n1,
 		FixedIn:     []database.FeatureVersion{f1, f3, f6, f7},
-		Severity:    types.Low,
+		Severity:    database.LowSeverity,
 		Description: "TestInsertVulnerabilityDescription1",
 		Link:        "TestInsertVulnerabilityLink1",
 		Metadata:    v1meta,
@@ -239,7 +232,7 @@ func TestInsertVulnerability(t *testing.T) {
 	// Update vulnerability.
 	v1.Description = "TestInsertVulnerabilityLink2"
 	v1.Link = "TestInsertVulnerabilityLink2"
-	v1.Severity = types.High
+	v1.Severity = database.HighSeverity
 	// Update f3 in f4, add fixed in f5, add fixed in f6 which already exists,
 	// removes fixed in f7 by adding f8 which is f7 but with MinVersion, and
 	// add fixed by f5 a second time (duplicated).
@@ -288,3 +281,16 @@ func equalsVuln(t *testing.T, expected, actual *database.Vulnerability) {
 		}
 	}
 }
+
+func TestStringComparison(t *testing.T) {
+	cmp := compareStringLists([]string{"a", "b", "b", "a"}, []string{"a", "c"})
+	assert.Len(t, cmp, 1)
+	assert.NotContains(t, cmp, "a")
+	assert.Contains(t, cmp, "b")
+
+	cmp = compareStringListsInBoth([]string{"a", "a", "b", "c"}, []string{"a", "c", "c"})
+	assert.Len(t, cmp, 2)
+	assert.NotContains(t, cmp, "b")
+	assert.Contains(t, cmp, "a")
+	assert.Contains(t, cmp, "c")
+}

database/severity.go

@@ -0,0 +1,134 @@
+// Copyright 2017 clair authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package database
+
+import (
+	"database/sql/driver"
+	"errors"
+	"strings"
+)
+
+// ErrFailedToParseSeverity is the error returned when a severity could not
+// be parsed from a string.
+var ErrFailedToParseSeverity = errors.New("failed to parse Severity from input")
+
+// Severity defines a standard scale for measuring the severity of a
+// vulnerability.
+type Severity string
+
+const (
+	// UnknownSeverity is either a security problem that has not been assigned to
+	// a priority yet or a priority that our system did not recognize.
+	UnknownSeverity Severity = "Unknown"
+
+	// NegligibleSeverity is technically a security problem, but is only
+	// theoretical in nature, requires a very special situation, has almost no
+	// install base, or does no real damage. These tend not to get backport from
+	// upstreams, and will likely not be included in security updates unless
+	// there is an easy fix and some other issue causes an update.
+	NegligibleSeverity Severity = "Negligible"
+
+	// LowSeverity is a security problem, but is hard to exploit due to
+	// environment, requires a user-assisted attack, a small install base, or
+	// does very little damage.  These tend to be included in security updates
+	// only when higher priority issues require an update, or if many low
+	// priority issues have built up.
+	LowSeverity Severity = "Low"
+
+	// MediumSeverity is a real security problem, and is exploitable for many
+	// people.  Includes network daemon denial of service attacks, cross-site
+	// scripting, and gaining user privileges.  Updates should be made soon for
+	// this priority of issue.
+	MediumSeverity Severity = "Medium"
+
+	// HighSeverity is a real problem, exploitable for many people in a default
+	// installation. Includes serious remote denial of services, local root
+	// privilege escalations, or data loss.
+	HighSeverity Severity = "High"
+
+	// CriticalSeverity is a world-burning problem, exploitable for nearly all
+	// people in a default installation of Linux. Includes remote root privilege
+	// escalations, or massive data loss.
+	CriticalSeverity Severity = "Critical"
+
+	// Defcon1Severity is a Critical problem which has been manually highlighted
+	// by the team. It requires an immediate attention.
+	Defcon1Severity Severity = "Defcon1"
+)
+
+// Severities lists all known severities, ordered from lowest to highest.
+var Severities = []Severity{
+	UnknownSeverity,
+	NegligibleSeverity,
+	LowSeverity,
+	MediumSeverity,
+	HighSeverity,
+	CriticalSeverity,
+	Defcon1Severity,
+}
+
+// NewSeverity attempts to parse a string into a standard Severity value.
+func NewSeverity(s string) (Severity, error) {
+	for _, ss := range Severities {
+		if strings.EqualFold(s, string(ss)) {
+			return ss, nil
+		}
+	}
+
+	return UnknownSeverity, ErrFailedToParseSeverity
+}
+
+// Compare determines the equality of two severities.
+//
+// If the severities are equal, returns 0.
+// If the receiever is less, returns -1.
+// If the receiver is greater, returns 1.
+func (s Severity) Compare(s2 Severity) int {
+	var i1, i2 int
+
+	for i1 = 0; i1 < len(Severities); i1 = i1 + 1 {
+		if s == Severities[i1] {
+			break
+		}
+	}
+	for i2 = 0; i2 < len(Severities); i2 = i2 + 1 {
+		if s2 == Severities[i2] {
+			break
+		}
+	}
+
+	return i1 - i2
+}
+
+// Scan implements the database/sql.Scanner interface.
+func (s *Severity) Scan(value interface{}) error {
+	val, ok := value.([]byte)
+	if !ok {
+		return errors.New("could not scan a Severity from a non-string input")
+	}
+
+	var err error
+	*s, err = NewSeverity(string(val))
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// Value implements the database/sql/driver.Valuer interface.
+func (s Severity) Value() (driver.Value, error) {
+	return string(s), nil
+}

database/severity_test.go

@@ -0,0 +1,35 @@
+// Copyright 2017 clair authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package database
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestCompareSeverity(t *testing.T) {
+	assert.Equal(t, MediumSeverity.Compare(MediumSeverity), 0, "Severity comparison failed")
+	assert.True(t, MediumSeverity.Compare(HighSeverity) < 0, "Severity comparison failed")
+	assert.True(t, CriticalSeverity.Compare(LowSeverity) > 0, "Severity comparison failed")
+}
+
+func TestParseSeverity(t *testing.T) {
+	_, err := NewSeverity("Test")
+	assert.Equal(t, ErrFailedToParseSeverity, err)
+
+	_, err = NewSeverity("Unknown")
+	assert.Nil(t, err)
+}

ext/featurefmt/apk/apk.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package apk implements a featurefmt.Lister for APK packages.
 package apk
 
 import (
@@ -21,21 +22,22 @@ import (
 	"github.com/coreos/pkg/capnslog"
 
 	"github.com/coreos/clair/database"
+	"github.com/coreos/clair/ext/featurefmt"
 	"github.com/coreos/clair/ext/versionfmt"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
-	"github.com/coreos/clair/worker/detectors"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
-var log = capnslog.NewPackageLogger("github.com/coreos/clair", "worker/detectors/packages")
+var log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/featurefmt/apk")
 
 func init() {
-	detectors.RegisterFeaturesDetector("apk", &detector{})
+	featurefmt.RegisterLister("apk", &lister{})
 }
 
-type detector struct{}
+type lister struct{}
 
-func (d *detector) Detect(data map[string][]byte) ([]database.FeatureVersion, error) {
+func (l lister) ListFeatures(files tarutil.FilesMap) ([]database.FeatureVersion, error) {
-	file, exists := data["lib/apk/db/installed"]
+	file, exists := files["lib/apk/db/installed"]
 	if !exists {
 		return []database.FeatureVersion{}, nil
 	}
@@ -83,6 +85,6 @@ func (d *detector) Detect(data map[string][]byte) ([]database.FeatureVersion, er
 	return pkgs, nil
 }
 
-func (d *detector) GetRequiredFiles() []string {
+func (l lister) RequiredFilenames() []string {
 	return []string{"lib/apk/db/installed"}
 }

ext/featurefmt/apk/apk_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -18,11 +18,12 @@ import (
 	"testing"
 
 	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/worker/detectors/feature"
+	"github.com/coreos/clair/ext/featurefmt"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
 func TestAPKFeatureDetection(t *testing.T) {
-	testData := []feature.TestData{
+	testData := []featurefmt.TestData{
 		{
 			FeatureVersions: []database.FeatureVersion{
 				{
@@ -70,10 +71,10 @@ func TestAPKFeatureDetection(t *testing.T) {
 					Version: "0.7-r0",
 				},
 			},
-			Data: map[string][]byte{
+			Files: tarutil.FilesMap{
-				"lib/apk/db/installed": feature.LoadFileForTest("apk/testdata/installed"),
+				"lib/apk/db/installed": featurefmt.LoadFileForTest("apk/testdata/installed"),
 			},
 		},
 	}
-	feature.TestDetector(t, &detector{}, testData)
+	featurefmt.TestLister(t, &lister{}, testData)
 }

ext/featurefmt/dpkg/dpkg.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package dpkg implements a featurefmt.Lister for dpkg packages.
 package dpkg
 
 import (
@@ -22,28 +23,27 @@ import (
 	"github.com/coreos/pkg/capnslog"
 
 	"github.com/coreos/clair/database"
+	"github.com/coreos/clair/ext/featurefmt"
 	"github.com/coreos/clair/ext/versionfmt"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
-	"github.com/coreos/clair/worker/detectors"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
 var (
-	log = capnslog.NewPackageLogger("github.com/coreos/clair", "worker/detectors/packages")
+	log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/featurefmt/dpkg")
 
 	dpkgSrcCaptureRegexp      = regexp.MustCompile(`Source: (?P<name>[^\s]*)( \((?P<version>.*)\))?`)
 	dpkgSrcCaptureRegexpNames = dpkgSrcCaptureRegexp.SubexpNames()
 )
 
-// DpkgFeaturesDetector implements FeaturesDetector and detects dpkg packages
+type lister struct{}
-type DpkgFeaturesDetector struct{}
 
 func init() {
-	detectors.RegisterFeaturesDetector("dpkg", &DpkgFeaturesDetector{})
+	featurefmt.RegisterLister("dpkg", &lister{})
 }
 
-// Detect detects packages using var/lib/dpkg/status from the input data
+func (l lister) ListFeatures(files tarutil.FilesMap) ([]database.FeatureVersion, error) {
-func (detector *DpkgFeaturesDetector) Detect(data map[string][]byte) ([]database.FeatureVersion, error) {
+	f, hasFile := files["var/lib/dpkg/status"]
-	f, hasFile := data["var/lib/dpkg/status"]
 	if !hasFile {
 		return []database.FeatureVersion{}, nil
 	}
@@ -116,8 +116,6 @@ func (detector *DpkgFeaturesDetector) Detect(data map[string][]byte) ([]database
 	return packages, nil
 }
 
-// GetRequiredFiles returns the list of files required for Detect, without
+func (l lister) RequiredFilenames() []string {
-// leading /
-func (detector *DpkgFeaturesDetector) GetRequiredFiles() []string {
 	return []string{"var/lib/dpkg/status"}
 }

ext/featurefmt/dpkg/dpkg_test.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -18,11 +18,12 @@ import (
 	"testing"
 
 	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/worker/detectors/feature"
+	"github.com/coreos/clair/ext/featurefmt"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
 func TestDpkgFeatureDetection(t *testing.T) {
-	testData := []feature.TestData{
+	testData := []featurefmt.TestData{
 		// Test an Ubuntu dpkg status file
 		{
 			FeatureVersions: []database.FeatureVersion{
@@ -40,11 +41,11 @@ func TestDpkgFeatureDetection(t *testing.T) {
 					Version: "5.1.1-12ubuntu1", // The version comes from the "Source:" line
 				},
 			},
-			Data: map[string][]byte{
+			Files: tarutil.FilesMap{
-				"var/lib/dpkg/status": feature.LoadFileForTest("dpkg/testdata/status"),
+				"var/lib/dpkg/status": featurefmt.LoadFileForTest("dpkg/testdata/status"),
 			},
 		},
 	}
 
-	feature.TestDetector(t, &DpkgFeaturesDetector{}, testData)
+	featurefmt.TestLister(t, &lister{}, testData)
 }

ext/featurefmt/driver.go

@@ -0,0 +1,126 @@
+// Copyright 2017 clair authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package featurefmt exposes functions to dynamically register methods for
+// determining the features present in an image layer.
+package featurefmt
+
+import (
+	"io/ioutil"
+	"path/filepath"
+	"runtime"
+	"sync"
+	"testing"
+
+	"github.com/coreos/clair/database"
+	"github.com/coreos/clair/pkg/tarutil"
+	"github.com/stretchr/testify/assert"
+)
+
+var (
+	listersM sync.RWMutex
+	listers  = make(map[string]Lister)
+)
+
+// Lister represents an ability to list the features present in an image layer.
+type Lister interface {
+	// ListFeatures produces a list of FeatureVersions present in an image layer.
+	ListFeatures(tarutil.FilesMap) ([]database.FeatureVersion, error)
+
+	// RequiredFilenames returns the list of files required to be in the FilesMap
+	// provided to the ListFeatures method.
+	//
+	// Filenames must not begin with "/".
+	RequiredFilenames() []string
+}
+
+// RegisterLister makes a Lister available by the provided name.
+//
+// If called twice with the same name, the name is blank, or if the provided
+// Lister is nil, this function panics.
+func RegisterLister(name string, l Lister) {
+	if name == "" {
+		panic("featurefmt: could not register a Lister with an empty name")
+	}
+	if l == nil {
+		panic("featurefmt: could not register a nil Lister")
+	}
+
+	listersM.Lock()
+	defer listersM.Unlock()
+
+	if _, dup := listers[name]; dup {
+		panic("featurefmt: RegisterLister called twice for " + name)
+	}
+
+	listers[name] = l
+}
+
+// ListFeatures produces the list of FeatureVersions in an image layer using
+// every registered Lister.
+func ListFeatures(files tarutil.FilesMap) ([]database.FeatureVersion, error) {
+	listersM.RLock()
+	defer listersM.RUnlock()
+
+	var totalFeatures []database.FeatureVersion
+	for _, lister := range listers {
+		features, err := lister.ListFeatures(files)
+		if err != nil {
+			return []database.FeatureVersion{}, err
+		}
+		totalFeatures = append(totalFeatures, features...)
+	}
+
+	return totalFeatures, nil
+}
+
+// RequiredFilenames returns the total list of files required for all
+// registered Listers.
+func RequiredFilenames() (files []string) {
+	listersM.RLock()
+	defer listersM.RUnlock()
+
+	for _, lister := range listers {
+		files = append(files, lister.RequiredFilenames()...)
+	}
+
+	return
+}
+
+// TestData represents the data used to test an implementation of Lister.
+type TestData struct {
+	Files           tarutil.FilesMap
+	FeatureVersions []database.FeatureVersion
+}
+
+// LoadFileForTest can be used in order to obtain the []byte contents of a file
+// that is meant to be used for test data.
+func LoadFileForTest(name string) []byte {
+	_, filename, _, _ := runtime.Caller(0)
+	d, _ := ioutil.ReadFile(filepath.Join(filepath.Dir(filename)) + "/" + name)
+	return d
+}
+
+// TestLister runs a Lister on each provided instance of TestData and asserts
+// the ouput to be equal to the expected output.
+func TestLister(t *testing.T, l Lister, testData []TestData) {
+	for _, td := range testData {
+		featureVersions, err := l.ListFeatures(td.Files)
+		if assert.Nil(t, err) && assert.Len(t, featureVersions, len(td.FeatureVersions)) {
+			for _, expectedFeatureVersion := range td.FeatureVersions {
+				assert.Contains(t, featureVersions, expectedFeatureVersion)
+			}
+		}
+	}
+}

ext/featurefmt/rpm/rpm.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,37 +12,36 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package rpm implements a featurefmt.Lister for rpm packages.
 package rpm
 
 import (
 	"bufio"
 	"io/ioutil"
 	"os"
+	"os/exec"
 	"strings"
 
 	"github.com/coreos/pkg/capnslog"
 
 	"github.com/coreos/clair/database"
+	"github.com/coreos/clair/ext/featurefmt"
 	"github.com/coreos/clair/ext/versionfmt"
 	"github.com/coreos/clair/ext/versionfmt/rpm"
-	"github.com/coreos/clair/utils"
+	"github.com/coreos/clair/pkg/commonerr"
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/tarutil"
-	"github.com/coreos/clair/worker/detectors"
 )
 
-var log = capnslog.NewPackageLogger("github.com/coreos/clair", "rpm")
+var log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/featurefmt/rpm")
 
-// RpmFeaturesDetector implements FeaturesDetector and detects rpm packages
+type lister struct{}
-// It requires the "rpm" binary to be in the PATH
-type RpmFeaturesDetector struct{}
 
 func init() {
-	detectors.RegisterFeaturesDetector("rpm", &RpmFeaturesDetector{})
+	featurefmt.RegisterLister("rpm", &lister{})
 }
 
-// Detect detects packages using var/lib/rpm/Packages from the input data
+func (l lister) ListFeatures(files tarutil.FilesMap) ([]database.FeatureVersion, error) {
-func (detector *RpmFeaturesDetector) Detect(data map[string][]byte) ([]database.FeatureVersion, error) {
+	f, hasFile := files["var/lib/rpm/Packages"]
-	f, hasFile := data["var/lib/rpm/Packages"]
 	if !hasFile {
 		return []database.FeatureVersion{}, nil
 	}
@@ -55,19 +54,17 @@ func (detector *RpmFeaturesDetector) Detect(data map[string][]byte) ([]database.
 	defer os.RemoveAll(tmpDir)
 	if err != nil {
 		log.Errorf("could not create temporary folder for RPM detection: %s", err)
-		return []database.FeatureVersion{}, cerrors.ErrFilesystem
+		return []database.FeatureVersion{}, commonerr.ErrFilesystem
 	}
 
 	err = ioutil.WriteFile(tmpDir+"/Packages", f, 0700)
 	if err != nil {
 		log.Errorf("could not create temporary file for RPM detection: %s", err)
-		return []database.FeatureVersion{}, cerrors.ErrFilesystem
+		return []database.FeatureVersion{}, commonerr.ErrFilesystem
 	}
 
-	// Query RPM
+	// Extract binary package names because RHSA refers to binary package names.
-	// We actually extract binary package names instead of source package names here because RHSA refers to package names
+	out, err := exec.Command("rpm", "--dbpath", tmpDir, "-qa", "--qf", "%{NAME} %{EPOCH}:%{VERSION}-%{RELEASE}\n").CombinedOutput()
-	// In the dpkg system, we extract the source instead
-	out, err := utils.Exec(tmpDir, "rpm", "--dbpath", tmpDir, "-qa", "--qf", "%{NAME} %{EPOCH}:%{VERSION}-%{RELEASE}\n")
 	if err != nil {
 		log.Errorf("could not query RPM: %s. output: %s", err, string(out))
 		// Do not bubble up because we probably won't be able to fix it,
@@ -116,8 +113,6 @@ func (detector *RpmFeaturesDetector) Detect(data map[string][]byte) ([]database.
 	return packages, nil
 }
 
-// GetRequiredFiles returns the list of files required for Detect, without
+func (l lister) RequiredFilenames() []string {
-// leading /
-func (detector *RpmFeaturesDetector) GetRequiredFiles() []string {
 	return []string{"var/lib/rpm/Packages"}
 }

ext/featurefmt/rpm/rpm_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -18,11 +18,12 @@ import (
 	"testing"
 
 	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/worker/detectors/feature"
+	"github.com/coreos/clair/ext/featurefmt"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
 func TestRpmFeatureDetection(t *testing.T) {
-	testData := []feature.TestData{
+	testData := []featurefmt.TestData{
 		// Test a CentOS 7 RPM database
 		// Memo: Use the following command on a RPM-based system to shrink a database: rpm -qa --qf "%{NAME}\n" |tail -n +3| xargs rpm -e --justdb
 		{
@@ -38,11 +39,11 @@ func TestRpmFeatureDetection(t *testing.T) {
 					Version: "3.2-18.el7",
 				},
 			},
-			Data: map[string][]byte{
+			Files: tarutil.FilesMap{
-				"var/lib/rpm/Packages": feature.LoadFileForTest("rpm/testdata/Packages"),
+				"var/lib/rpm/Packages": featurefmt.LoadFileForTest("rpm/testdata/Packages"),
 			},
 		},
 	}
 
-	feature.TestDetector(t, &RpmFeaturesDetector{}, testData)
+	featurefmt.TestLister(t, &lister{}, testData)
 }

ext/featurens/alpinerelease/alpinerelease.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,6 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package alpinerelease implements a featurens.Detector for Alpine Linux based
+// container image layers.
 package alpinerelease
 
 import (
@@ -21,8 +23,9 @@ import (
 	"strings"
 
 	"github.com/coreos/clair/database"
+	"github.com/coreos/clair/ext/featurens"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
-	"github.com/coreos/clair/worker/detectors"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
 const (
@@ -33,15 +36,13 @@ const (
 var versionRegexp = regexp.MustCompile(`^(\d)+\.(\d)+\.(\d)+$`)
 
 func init() {
-	detectors.RegisterNamespaceDetector("alpine-release", &detector{})
+	featurens.RegisterDetector("alpine-release", &detector{})
 }
 
-// detector implements NamespaceDetector by reading the current version of
-// Alpine Linux from /etc/alpine-release.
 type detector struct{}
 
-func (d *detector) Detect(data map[string][]byte) *database.Namespace {
+func (d detector) Detect(files tarutil.FilesMap) (*database.Namespace, error) {
-	file, exists := data[alpineReleasePath]
+	file, exists := files[alpineReleasePath]
 	if exists {
 		scanner := bufio.NewScanner(bytes.NewBuffer(file))
 		for scanner.Scan() {
@@ -52,14 +53,14 @@ func (d *detector) Detect(data map[string][]byte) *database.Namespace {
 				return &database.Namespace{
 					Name:          osName + ":" + "v" + versionNumbers[0] + "." + versionNumbers[1],
 					VersionFormat: dpkg.ParserName,
-				}
+				}, nil
 			}
 		}
 	}
 
-	return nil
+	return nil, nil
 }
 
-func (d *detector) GetRequiredFiles() []string {
+func (d detector) RequiredFilenames() []string {
 	return []string{alpineReleasePath}
 }

ext/featurens/alpinerelease/alpinerelease_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -18,34 +18,35 @@ import (
 	"testing"
 
 	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/worker/detectors/namespace"
+	"github.com/coreos/clair/ext/featurens"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
-func TestAlpineReleaseNamespaceDetection(t *testing.T) {
+func TestDetector(t *testing.T) {
-	testData := []namespace.TestData{
+	testData := []featurens.TestData{
 		{
 			ExpectedNamespace: &database.Namespace{Name: "alpine:v3.3"},
-			Data:              map[string][]byte{"etc/alpine-release": []byte(`3.3.4`)},
+			Files:             tarutil.FilesMap{"etc/alpine-release": []byte(`3.3.4`)},
 		},
 		{
 			ExpectedNamespace: &database.Namespace{Name: "alpine:v3.4"},
-			Data:              map[string][]byte{"etc/alpine-release": []byte(`3.4.0`)},
+			Files:             tarutil.FilesMap{"etc/alpine-release": []byte(`3.4.0`)},
 		},
 		{
 			ExpectedNamespace: &database.Namespace{Name: "alpine:v0.3"},
-			Data:              map[string][]byte{"etc/alpine-release": []byte(`0.3.4`)},
+			Files:             tarutil.FilesMap{"etc/alpine-release": []byte(`0.3.4`)},
 		},
 		{
 			ExpectedNamespace: &database.Namespace{Name: "alpine:v0.3"},
-			Data: map[string][]byte{"etc/alpine-release": []byte(`
+			Files: tarutil.FilesMap{"etc/alpine-release": []byte(`
 0.3.4
 `)},
 		},
 		{
 			ExpectedNamespace: nil,
-			Data:              map[string][]byte{},
+			Files:             tarutil.FilesMap{},
 		},
 	}
 
-	namespace.TestDetector(t, &detector{}, testData)
+	featurens.TestDetector(t, &detector{}, testData)
 }

ext/featurens/aptsources/aptsources.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,6 +12,11 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package aptsources implements a featurens.Detector for apt based container
+// image layers.
+//
+// This detector is necessary to determine the precise Debian version when it
+// is an unstable version for instance.
 package aptsources
 
 import (
@@ -19,25 +24,21 @@ import (
 	"strings"
 
 	"github.com/coreos/clair/database"
+	"github.com/coreos/clair/ext/featurens"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
-	"github.com/coreos/clair/worker/detectors"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
-// AptSourcesNamespaceDetector implements NamespaceDetector and detects the Namespace from the
+type detector struct{}
-// /etc/apt/sources.list file.
-//
-// This detector is necessary to determine precise Debian version when it is
-// an unstable version for instance.
-type AptSourcesNamespaceDetector struct{}
 
 func init() {
-	detectors.RegisterNamespaceDetector("apt-sources", &AptSourcesNamespaceDetector{})
+	featurens.RegisterDetector("apt-sources", &detector{})
 }
 
-func (detector *AptSourcesNamespaceDetector) Detect(data map[string][]byte) *database.Namespace {
+func (d detector) Detect(files tarutil.FilesMap) (*database.Namespace, error) {
-	f, hasFile := data["etc/apt/sources.list"]
+	f, hasFile := files["etc/apt/sources.list"]
 	if !hasFile {
-		return nil
+		return nil, nil
 	}
 
 	var OS, version string
@@ -79,11 +80,11 @@ func (detector *AptSourcesNamespaceDetector) Detect(data map[string][]byte) *dat
 		return &database.Namespace{
 			Name:          OS + ":" + version,
 			VersionFormat: dpkg.ParserName,
-		}
+		}, nil
 	}
-	return nil
+	return nil, nil
 }
 
-func (detector *AptSourcesNamespaceDetector) GetRequiredFiles() []string {
+func (d detector) RequiredFilenames() []string {
 	return []string{"etc/apt/sources.list"}
 }

ext/featurens/aptsources/aptsources_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -18,14 +18,15 @@ import (
 	"testing"
 
 	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/worker/detectors/namespace"
+	"github.com/coreos/clair/ext/featurens"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
-func TestAptSourcesNamespaceDetector(t *testing.T) {
+func TestDetector(t *testing.T) {
-	testData := []namespace.TestData{
+	testData := []featurens.TestData{
 		{
 			ExpectedNamespace: &database.Namespace{Name: "debian:unstable"},
-			Data: map[string][]byte{
+			Files: tarutil.FilesMap{
 				"etc/os-release": []byte(
 					`PRETTY_NAME="Debian GNU/Linux stretch/sid"
 NAME="Debian GNU/Linux"
@@ -38,5 +39,5 @@ BUG_REPORT_URL="https://bugs.debian.org/"`),
 		},
 	}
 
-	namespace.TestDetector(t, &AptSourcesNamespaceDetector{}, testData)
+	featurens.TestDetector(t, &detector{}, testData)
 }

ext/featurens/driver.go

@@ -0,0 +1,127 @@
+// Copyright 2017 clair authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package featurens exposes functions to dynamically register methods for
+// determining a namespace for features present in an image layer.
+package featurens
+
+import (
+	"sync"
+	"testing"
+
+	"github.com/coreos/pkg/capnslog"
+	"github.com/stretchr/testify/assert"
+
+	"github.com/coreos/clair/database"
+	"github.com/coreos/clair/pkg/tarutil"
+)
+
+var (
+	log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/featurens")
+
+	detectorsM sync.RWMutex
+	detectors  = make(map[string]Detector)
+)
+
+// Detector represents an ability to detect a namespace used for organizing
+// features present in an image layer.
+type Detector interface {
+	// Detect attempts to determine a Namespace from a FilesMap of an image
+	// layer.
+	Detect(tarutil.FilesMap) (*database.Namespace, error)
+
+	// RequiredFilenames returns the list of files required to be in the FilesMap
+	// provided to the Detect method.
+	//
+	// Filenames must not begin with "/".
+	RequiredFilenames() []string
+}
+
+// RegisterDetector makes a detector available by the provided name.
+//
+// If called twice with the same name, the name is blank, or if the provided
+// Detector is nil, this function panics.
+func RegisterDetector(name string, d Detector) {
+	if name == "" {
+		panic("namespace: could not register a Detector with an empty name")
+	}
+	if d == nil {
+		panic("namespace: could not register a nil Detector")
+	}
+
+	detectorsM.Lock()
+	defer detectorsM.Unlock()
+
+	if _, dup := detectors[name]; dup {
+		panic("namespace: RegisterDetector called twice for " + name)
+	}
+
+	detectors[name] = d
+}
+
+// Detect iterators through all registered Detectors and returns the first
+// non-nil detected namespace.
+func Detect(files tarutil.FilesMap) (*database.Namespace, error) {
+	detectorsM.RLock()
+	defer detectorsM.RUnlock()
+
+	for name, detector := range detectors {
+		namespace, err := detector.Detect(files)
+		if err != nil {
+			log.Warningf("failed while attempting to detect namespace %s: %s", name, err)
+			return nil, err
+		}
+
+		if namespace != nil {
+			log.Debugf("detected namespace %s: %#v", name, namespace)
+			return namespace, nil
+		}
+	}
+
+	return nil, nil
+}
+
+// RequiredFilenames returns the total list of files required for all
+// registered Detectors.
+func RequiredFilenames() (files []string) {
+	detectorsM.RLock()
+	defer detectorsM.RUnlock()
+
+	for _, detector := range detectors {
+		files = append(files, detector.RequiredFilenames()...)
+	}
+
+	return
+}
+
+// TestData represents the data used to test an implementation of Detector.
+type TestData struct {
+	Files             tarutil.FilesMap
+	ExpectedNamespace *database.Namespace
+}
+
+// TestDetector runs a Detector on each provided instance of TestData and
+// asserts the output to be equal to the expected output.
+func TestDetector(t *testing.T, d Detector, testData []TestData) {
+	for _, td := range testData {
+		namespace, err := d.Detect(td.Files)
+		assert.Nil(t, err)
+
+		if namespace == nil {
+			assert.Equal(t, td.ExpectedNamespace, namespace)
+		} else {
+			assert.Equal(t, td.ExpectedNamespace.Name, namespace.Name)
+		}
+	}
+}

ext/featurens/lsbrelease/lsbrelease.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,6 +12,10 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package lsbrelease implements a featurens.Detector for container image
+// layers containing an lsb-release file.
+//
+// This detector is necessary for detecting Ubuntu Precise.
 package lsbrelease
 
 import (
@@ -20,9 +24,10 @@ import (
 	"strings"
 
 	"github.com/coreos/clair/database"
+	"github.com/coreos/clair/ext/featurens"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
 	"github.com/coreos/clair/ext/versionfmt/rpm"
-	"github.com/coreos/clair/worker/detectors"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
 var (
@@ -30,20 +35,16 @@ var (
 	lsbReleaseVersionRegexp = regexp.MustCompile(`^DISTRIB_RELEASE=(.*)`)
 )
 
-// LsbReleaseNamespaceDetector implements NamespaceDetector and detects the
+type detector struct{}
-// Namespace from the /etc/lsb-release file.
-//
-// This detector is necessary for Ubuntu Precise.
-type LsbReleaseNamespaceDetector struct{}
 
 func init() {
-	detectors.RegisterNamespaceDetector("lsb-release", &LsbReleaseNamespaceDetector{})
+	featurens.RegisterDetector("lsb-release", &detector{})
 }
 
-func (detector *LsbReleaseNamespaceDetector) Detect(data map[string][]byte) *database.Namespace {
+func (d detector) Detect(files tarutil.FilesMap) (*database.Namespace, error) {
-	f, hasFile := data["etc/lsb-release"]
+	f, hasFile := files["etc/lsb-release"]
 	if !hasFile {
-		return nil
+		return nil, nil
 	}
 
 	var OS, version string
@@ -79,19 +80,19 @@ func (detector *LsbReleaseNamespaceDetector) Detect(data map[string][]byte) *dat
 	case "centos", "rhel", "fedora", "amzn", "ol", "oracle":
 		versionFormat = rpm.ParserName
 	default:
-		return nil
+		return nil, nil
 	}
 
 	if OS != "" && version != "" {
 		return &database.Namespace{
 			Name:          OS + ":" + version,
 			VersionFormat: versionFormat,
-		}
+		}, nil
 	}
-	return nil
+
+	return nil, nil
 }
 
-// GetRequiredFiles returns the list of files that are required for Detect()
+func (d *detector) RequiredFilenames() []string {
-func (detector *LsbReleaseNamespaceDetector) GetRequiredFiles() []string {
 	return []string{"etc/lsb-release"}
 }

ext/featurens/lsbrelease/lsbrelease_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -18,14 +18,15 @@ import (
 	"testing"
 
 	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/worker/detectors/namespace"
+	"github.com/coreos/clair/ext/featurens"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
-func TestLsbReleaseNamespaceDetector(t *testing.T) {
+func TestDetector(t *testing.T) {
-	testData := []namespace.TestData{
+	testData := []featurens.TestData{
 		{
 			ExpectedNamespace: &database.Namespace{Name: "ubuntu:12.04"},
-			Data: map[string][]byte{
+			Files: tarutil.FilesMap{
 				"etc/lsb-release": []byte(
 					`DISTRIB_ID=Ubuntu
 DISTRIB_RELEASE=12.04
@@ -35,7 +36,7 @@ DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS"`),
 		},
 		{ // We don't care about the minor version of Debian
 			ExpectedNamespace: &database.Namespace{Name: "debian:7"},
-			Data: map[string][]byte{
+			Files: tarutil.FilesMap{
 				"etc/lsb-release": []byte(
 					`DISTRIB_ID=Debian
 DISTRIB_RELEASE=7.1
@@ -45,5 +46,5 @@ DISTRIB_DESCRIPTION="Debian 7.1"`),
 		},
 	}
 
-	namespace.TestDetector(t, &LsbReleaseNamespaceDetector{}, testData)
+	featurens.TestDetector(t, &detector{}, testData)
 }

ext/featurens/osrelease/osrelease.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,6 +12,10 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package osrelease implements a featurens.Detector for container image
+// layers containing an os-release file.
+//
+// This detector is typically useful for detecting Debian or Ubuntu.
 package osrelease
 
 import (
@@ -20,40 +24,41 @@ import (
 	"strings"
 
 	"github.com/coreos/clair/database"
+	"github.com/coreos/clair/ext/featurens"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
 	"github.com/coreos/clair/ext/versionfmt/rpm"
-	"github.com/coreos/clair/worker/detectors"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
 var (
-	//log = capnslog.NewPackageLogger("github.com/coreos/clair", "worker/detectors/namespace/osrelease")
-
 	osReleaseOSRegexp      = regexp.MustCompile(`^ID=(.*)`)
 	osReleaseVersionRegexp = regexp.MustCompile(`^VERSION_ID=(.*)`)
+
+	// blacklistFilenames are files that should exclude this detector.
+	blacklistFilenames = []string{
+		"etc/oracle-release",
+		"etc/redhat-release",
+		"usr/lib/centos-release",
+	}
 )
 
-// OsReleaseNamespaceDetector implements NamespaceDetector and detects the OS from the
+type detector struct{}
-// /etc/os-release and usr/lib/os-release files.
-type OsReleaseNamespaceDetector struct{}
 
 func init() {
-	detectors.RegisterNamespaceDetector("os-release", &OsReleaseNamespaceDetector{})
+	featurens.RegisterDetector("os-release", &detector{})
 }
 
-// Detect tries to detect OS/Version using "/etc/os-release" and "/usr/lib/os-release"
+func (d detector) Detect(files tarutil.FilesMap) (*database.Namespace, error) {
-// Typically for Debian / Ubuntu
-// /etc/debian_version can't be used, it does not make any difference between testing and unstable, it returns stretch/sid
-func (detector *OsReleaseNamespaceDetector) Detect(data map[string][]byte) *database.Namespace {
 	var OS, version string
 
-	for _, filePath := range detector.getExcludeFiles() {
+	for _, filePath := range blacklistFilenames {
-		if _, hasFile := data[filePath]; hasFile {
+		if _, hasFile := files[filePath]; hasFile {
-			return nil
+			return nil, nil
 		}
 	}
 
-	for _, filePath := range detector.GetRequiredFiles() {
+	for _, filePath := range d.RequiredFilenames() {
-		f, hasFile := data[filePath]
+		f, hasFile := files[filePath]
 		if !hasFile {
 			continue
 		}
@@ -82,24 +87,18 @@ func (detector *OsReleaseNamespaceDetector) Detect(data map[string][]byte) *data
 	case "centos", "rhel", "fedora", "amzn", "ol", "oracle":
 		versionFormat = rpm.ParserName
 	default:
-		return nil
+		return nil, nil
 	}
 
 	if OS != "" && version != "" {
 		return &database.Namespace{
 			Name:          OS + ":" + version,
 			VersionFormat: versionFormat,
-		}
+		}, nil
 	}
-	return nil
+	return nil, nil
 }
 
-// GetRequiredFiles returns the list of files that are required for Detect()
+func (d detector) RequiredFilenames() []string {
-func (detector *OsReleaseNamespaceDetector) GetRequiredFiles() []string {
 	return []string{"etc/os-release", "usr/lib/os-release"}
 }
-
-// getExcludeFiles returns the list of files that are ought to exclude this detector from Detect()
-func (detector *OsReleaseNamespaceDetector) getExcludeFiles() []string {
-	return []string{"etc/oracle-release", "etc/redhat-release", "usr/lib/centos-release"}
-}

ext/featurens/osrelease/osrelease_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -18,14 +18,15 @@ import (
 	"testing"
 
 	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/worker/detectors/namespace"
+	"github.com/coreos/clair/ext/featurens"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
-func TestOsReleaseNamespaceDetector(t *testing.T) {
+func TestDetector(t *testing.T) {
-	testData := []namespace.TestData{
+	testData := []featurens.TestData{
 		{
 			ExpectedNamespace: &database.Namespace{Name: "debian:8"},
-			Data: map[string][]byte{
+			Files: tarutil.FilesMap{
 				"etc/os-release": []byte(
 					`PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
 NAME="Debian GNU/Linux"
@@ -39,7 +40,7 @@ BUG_REPORT_URL="https://bugs.debian.org/"`),
 		},
 		{
 			ExpectedNamespace: &database.Namespace{Name: "ubuntu:15.10"},
-			Data: map[string][]byte{
+			Files: tarutil.FilesMap{
 				"etc/os-release": []byte(
 					`NAME="Ubuntu"
 VERSION="15.10 (Wily Werewolf)"
@@ -54,7 +55,7 @@ BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"`),
 		},
 		{ // Doesn't have quotes around VERSION_ID
 			ExpectedNamespace: &database.Namespace{Name: "fedora:20"},
-			Data: map[string][]byte{
+			Files: tarutil.FilesMap{
 				"etc/os-release": []byte(
 					`NAME=Fedora
 VERSION="20 (Heisenbug)"
@@ -73,5 +74,5 @@ REDHAT_SUPPORT_PRODUCT_VERSION=20`),
 		},
 	}
 
-	namespace.TestDetector(t, &OsReleaseNamespaceDetector{}, testData)
+	featurens.TestDetector(t, &detector{}, testData)
 }

ext/featurens/redhatrelease/redhatrelease.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,6 +12,11 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package redhatrelease implements a featurens.Detector for container image
+// layers containing an redhat-release-like files.
+//
+// This detector is typically useful for detecting CentOS and Red-Hat like
+// systems.
 package redhatrelease
 
 import (
@@ -19,77 +24,64 @@ import (
 	"strings"
 
 	"github.com/coreos/clair/database"
+	"github.com/coreos/clair/ext/featurens"
 	"github.com/coreos/clair/ext/versionfmt/rpm"
-	"github.com/coreos/clair/worker/detectors"
+	"github.com/coreos/clair/pkg/tarutil"
-	"github.com/coreos/pkg/capnslog"
 )
 
 var (
-	log = capnslog.NewPackageLogger("github.com/coreos/clair", "worker/detectors/namespace/redhatrelease")
-
 	oracleReleaseRegexp = regexp.MustCompile(`(?P<os>[^\s]*) (Linux Server release) (?P<version>[\d]+)`)
 	centosReleaseRegexp = regexp.MustCompile(`(?P<os>[^\s]*) (Linux release|release) (?P<version>[\d]+)`)
 	redhatReleaseRegexp = regexp.MustCompile(`(?P<os>Red Hat Enterprise Linux) (Client release|Server release|Workstation release) (?P<version>[\d]+)`)
 )
 
-// RedhatReleaseNamespaceDetector implements NamespaceDetector and detects the OS from the
+type detector struct{}
-// /etc/oracle-release, /etc/centos-release, /etc/redhat-release and /etc/system-release files.
-//
-// Typically for CentOS and Red-Hat like systems
-// eg. CentOS release 5.11 (Final)
-// eg. CentOS release 6.6 (Final)
-// eg. CentOS Linux release 7.1.1503 (Core)
-// eg. Oracle Linux Server release 7.3
-// eg. Red Hat Enterprise Linux Server release 7.2 (Maipo)
-type RedhatReleaseNamespaceDetector struct{}
 
 func init() {
-	detectors.RegisterNamespaceDetector("redhat-release", &RedhatReleaseNamespaceDetector{})
+	featurens.RegisterDetector("redhat-release", &detector{})
 }
 
-func (detector *RedhatReleaseNamespaceDetector) Detect(data map[string][]byte) *database.Namespace {
+func (d detector) Detect(files tarutil.FilesMap) (*database.Namespace, error) {
-	for _, filePath := range detector.GetRequiredFiles() {
+	for _, filePath := range d.RequiredFilenames() {
-		f, hasFile := data[filePath]
+		f, hasFile := files[filePath]
 		if !hasFile {
 			continue
 		}
 
 		var r []string
 
-		// try for Oracle Linux
+		// Attempt to match Oracle Linux.
 		r = oracleReleaseRegexp.FindStringSubmatch(string(f))
 		if len(r) == 4 {
 			return &database.Namespace{
 				Name:          strings.ToLower(r[1]) + ":" + r[3],
 				VersionFormat: rpm.ParserName,
-			}
+			}, nil
 		}
 
-		// try for RHEL
+		// Attempt to match RHEL.
 		r = redhatReleaseRegexp.FindStringSubmatch(string(f))
 		if len(r) == 4 {
-			// TODO(vbatts) this is a hack until https://github.com/coreos/clair/pull/193
+			// TODO(vbatts): this is a hack until https://github.com/coreos/clair/pull/193
 			return &database.Namespace{
 				Name:          "centos" + ":" + r[3],
 				VersionFormat: rpm.ParserName,
-			}
+			}, nil
 		}
 
-		// then try centos first
+		// Atempt to match CentOS.
 		r = centosReleaseRegexp.FindStringSubmatch(string(f))
 		if len(r) == 4 {
 			return &database.Namespace{
 				Name:          strings.ToLower(r[1]) + ":" + r[3],
 				VersionFormat: rpm.ParserName,
-			}
+			}, nil
 		}
-
 	}
 
-	return nil
+	return nil, nil
 }
 
-// GetRequiredFiles returns the list of files that are required for Detect()
+func (d detector) RequiredFilenames() []string {
-func (detector *RedhatReleaseNamespaceDetector) GetRequiredFiles() []string {
 	return []string{"etc/oracle-release", "etc/centos-release", "etc/redhat-release", "etc/system-release"}
 }

ext/featurens/redhatrelease/redhatrelease_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -18,36 +18,37 @@ import (
 	"testing"
 
 	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/worker/detectors/namespace"
+	"github.com/coreos/clair/ext/featurens"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
-func TestRedhatReleaseNamespaceDetector(t *testing.T) {
+func TestDetector(t *testing.T) {
-	testData := []namespace.TestData{
+	testData := []featurens.TestData{
 		{
 			ExpectedNamespace: &database.Namespace{Name: "oracle:6"},
-			Data: map[string][]byte{
+			Files: tarutil.FilesMap{
 				"etc/oracle-release": []byte(`Oracle Linux Server release 6.8`),
 			},
 		},
 		{
 			ExpectedNamespace: &database.Namespace{Name: "oracle:7"},
-			Data: map[string][]byte{
+			Files: tarutil.FilesMap{
 				"etc/oracle-release": []byte(`Oracle Linux Server release 7.2`),
 			},
 		},
 		{
 			ExpectedNamespace: &database.Namespace{Name: "centos:6"},
-			Data: map[string][]byte{
+			Files: tarutil.FilesMap{
 				"etc/centos-release": []byte(`CentOS release 6.6 (Final)`),
 			},
 		},
 		{
 			ExpectedNamespace: &database.Namespace{Name: "centos:7"},
-			Data: map[string][]byte{
+			Files: tarutil.FilesMap{
 				"etc/system-release": []byte(`CentOS Linux release 7.1.1503 (Core)`),
 			},
 		},
 	}
 
-	namespace.TestDetector(t, &RedhatReleaseNamespaceDetector{}, testData)
+	featurens.TestDetector(t, &detector{}, testData)
 }

ext/imagefmt/aci/aci.go

@@ -0,0 +1,42 @@
+// Copyright 2017 clair authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package aci implements an imagefmt.Extractor for appc formatted container
+// image layers.
+package aci
+
+import (
+	"io"
+	"path/filepath"
+
+	"github.com/coreos/clair/ext/imagefmt"
+	"github.com/coreos/clair/pkg/tarutil"
+)
+
+type format struct{}
+
+func init() {
+	imagefmt.RegisterExtractor("aci", &format{})
+}
+
+func (f format) ExtractFiles(layerReader io.ReadCloser, toExtract []string) (tarutil.FilesMap, error) {
+	// All contents are inside a "rootfs" directory, so this needs to be
+	// prepended to each filename.
+	var filenames []string
+	for _, filename := range toExtract {
+		filenames = append(filenames, filepath.Join("rootfs/", filename))
+	}
+
+	return tarutil.ExtractFiles(layerReader, filenames)
+}

ext/imagefmt/docker/docker.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,21 +12,23 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package types
+// Package docker implements an imagefmt.Extractor for docker formatted
+// container image layers.
+package docker
 
 import (
-	"testing"
+	"io"
 
-	"github.com/stretchr/testify/assert"
+	"github.com/coreos/clair/ext/imagefmt"
+	"github.com/coreos/clair/pkg/tarutil"
 )
 
-func TestComparePriority(t *testing.T) {
+type format struct{}
-	assert.Equal(t, Medium.Compare(Medium), 0, "Priority comparison failed")
+
-	assert.True(t, Medium.Compare(High) < 0, "Priority comparison failed")
+func init() {
-	assert.True(t, Critical.Compare(Low) > 0, "Priority comparison failed")
+	imagefmt.RegisterExtractor("docker", &format{})
 }
 
-func TestIsValid(t *testing.T) {
+func (f format) ExtractFiles(layerReader io.ReadCloser, toExtract []string) (tarutil.FilesMap, error) {
-	assert.False(t, Priority("Test").IsValid())
+	return tarutil.ExtractFiles(layerReader, toExtract)
-	assert.True(t, Unknown.IsValid())
 }

ext/imagefmt/driver.go

@@ -0,0 +1,150 @@
+// Copyright 2017 clair authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package notification fetches notifications from the database and informs the
+// specified remote handler about their existences, inviting the third party to
+// actively query the API about it.
+
+// Package imagefmt exposes functions to dynamically register methods to
+// detect different types of container image formats.
+package imagefmt
+
+import (
+	"fmt"
+	"io"
+	"math"
+	"net/http"
+	"os"
+	"strings"
+	"sync"
+
+	"github.com/coreos/clair/pkg/commonerr"
+	"github.com/coreos/clair/pkg/tarutil"
+	"github.com/coreos/pkg/capnslog"
+)
+
+var (
+	// ErrCouldNotFindLayer is returned when we could not download or open the layer file.
+	ErrCouldNotFindLayer = commonerr.NewBadRequestError("could not find layer")
+
+	log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/imagefmt")
+
+	extractorsM sync.RWMutex
+	extractors  = make(map[string]Extractor)
+)
+
+// Extractor represents an ability to extract files from a particular container
+// image format.
+type Extractor interface {
+	// ExtractFiles produces a tarutil.FilesMap from a image layer.
+	ExtractFiles(layer io.ReadCloser, filenames []string) (tarutil.FilesMap, error)
+}
+
+// RegisterExtractor makes an extractor available by the provided name.
+//
+// If called twice with the same name, the name is blank, or if the provided
+// Extractor is nil, this function panics.
+func RegisterExtractor(name string, d Extractor) {
+	extractorsM.Lock()
+	defer extractorsM.Unlock()
+
+	if name == "" {
+		panic("imagefmt: could not register an Extractor with an empty name")
+	}
+
+	if d == nil {
+		panic("imagefmt: could not register a nil Extractor")
+	}
+
+	// Enforce lowercase names, so that they can be reliably be found in a map.
+	name = strings.ToLower(name)
+
+	if _, dup := extractors[name]; dup {
+		panic("imagefmt: RegisterExtractor called twice for " + name)
+	}
+
+	extractors[name] = d
+}
+
+// Extractors returns the list of the registered extractors.
+func Extractors() map[string]Extractor {
+	extractorsM.RLock()
+	defer extractorsM.RUnlock()
+
+	ret := make(map[string]Extractor)
+	for k, v := range extractors {
+		ret[k] = v
+	}
+
+	return ret
+}
+
+// UnregisterExtractor removes a Extractor with a particular name from the list.
+func UnregisterExtractor(name string) {
+	extractorsM.Lock()
+	defer extractorsM.Unlock()
+	delete(extractors, name)
+}
+
+// Extract streams an image layer from disk or over HTTP, determines the
+// image format, then extracts the files specified.
+func Extract(format, path string, headers map[string]string, toExtract []string) (tarutil.FilesMap, error) {
+	var layerReader io.ReadCloser
+	if strings.HasPrefix(path, "http://") || strings.HasPrefix(path, "https://") {
+		// Create a new HTTP request object.
+		request, err := http.NewRequest("GET", path, nil)
+		if err != nil {
+			return nil, ErrCouldNotFindLayer
+		}
+
+		// Set any provided HTTP Headers.
+		if headers != nil {
+			for k, v := range headers {
+				request.Header.Set(k, v)
+			}
+		}
+
+		// Send the request and handle the response.
+		r, err := http.DefaultClient.Do(request)
+		if err != nil {
+			log.Warningf("could not download layer: %s", err)
+			return nil, ErrCouldNotFindLayer
+		}
+
+		// Fail if we don't receive a 2xx HTTP status code.
+		if math.Floor(float64(r.StatusCode/100)) != 2 {
+			log.Warningf("could not download layer: got status code %d, expected 2XX", r.StatusCode)
+			return nil, ErrCouldNotFindLayer
+		}
+
+		layerReader = r.Body
+	} else {
+		var err error
+		layerReader, err = os.Open(path)
+		if err != nil {
+			return nil, ErrCouldNotFindLayer
+		}
+	}
+	defer layerReader.Close()
+
+	if extractor, exists := Extractors()[strings.ToLower(format)]; exists {
+		files, err := extractor.ExtractFiles(layerReader, toExtract)
+		if err != nil {
+			return nil, err
+		}
+		return files, nil
+	}
+
+	return nil, commonerr.NewBadRequestError(fmt.Sprintf("unsupported image format '%s'", format))
+}

ext/notification/driver.go

@@ -0,0 +1,99 @@
+// Copyright 2017 clair authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package notification fetches notifications from the database and informs the
+// specified remote handler about their existences, inviting the third party to
+// actively query the API about it.
+
+// Package notification exposes functions to dynamically register methods to
+// deliver notifications from the Clair database.
+package notification
+
+import (
+	"sync"
+	"time"
+
+	"github.com/coreos/pkg/capnslog"
+
+	"github.com/coreos/clair/database"
+)
+
+var (
+	log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/notification")
+
+	sendersM sync.RWMutex
+	senders  = make(map[string]Sender)
+)
+
+// Config is the configuration for the Notifier service and its registered
+// notifiers.
+type Config struct {
+	Attempts         int
+	RenotifyInterval time.Duration
+	Params           map[string]interface{} `yaml:",inline"`
+}
+
+// Sender represents anything that can transmit notifications.
+type Sender interface {
+	// Configure attempts to initialize the notifier with the provided configuration.
+	// It returns whether the notifier is enabled or not.
+	Configure(*Config) (bool, error)
+
+	// Send informs the existence of the specified notification.
+	Send(notification database.VulnerabilityNotification) error
+}
+
+// RegisterSender makes a Sender available by the provided name.
+//
+// If called twice with the same name, the name is blank, or if the provided
+// Sender is nil, this function panics.
+func RegisterSender(name string, s Sender) {
+	if name == "" {
+		panic("notification: could not register a Sender with an empty name")
+	}
+
+	if s == nil {
+		panic("notification: could not register a nil Sender")
+	}
+
+	sendersM.Lock()
+	defer sendersM.Unlock()
+
+	if _, dup := senders[name]; dup {
+		panic("notification: RegisterSender called twice for " + name)
+	}
+
+	senders[name] = s
+}
+
+// Senders returns the list of the registered Senders.
+func Senders() map[string]Sender {
+	sendersM.RLock()
+	defer sendersM.RUnlock()
+
+	ret := make(map[string]Sender)
+	for k, v := range senders {
+		ret[k] = v
+	}
+
+	return ret
+}
+
+// UnregisterSender removes a Sender with a particular name from the list.
+func UnregisterSender(name string) {
+	sendersM.Lock()
+	defer sendersM.Unlock()
+
+	delete(senders, name)
+}

ext/notification/webhook/webhook.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,8 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-// Package notifiers implements several kinds of notifier.Notifier
+// Package webhook implements a notification sender for HTTP JSON webhooks.
-package notifiers
+package webhook
 
 import (
 	"bytes"
@@ -29,21 +29,19 @@ import (
 
 	"gopkg.in/yaml.v2"
 
-	"github.com/coreos/clair/config"
 	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/notifier"
+	"github.com/coreos/clair/ext/notification"
 )
 
 const timeout = 5 * time.Second
 
-// A WebhookNotifier dispatches notifications to a webhook endpoint.
+type sender struct {
-type WebhookNotifier struct {
 	endpoint string
 	client   *http.Client
 }
 
-// A WebhookNotifierConfiguration represents the configuration of a WebhookNotifier.
+// Config represents the configuration of a Webhook Sender.
-type WebhookNotifierConfiguration struct {
+type Config struct {
 	Endpoint   string
 	ServerName string
 	CertFile   string
@@ -53,12 +51,12 @@ type WebhookNotifierConfiguration struct {
 }
 
 func init() {
-	notifier.RegisterNotifier("webhook", &WebhookNotifier{})
+	notification.RegisterSender("webhook", &sender{})
 }
 
-func (h *WebhookNotifier) Configure(config *config.NotifierConfig) (bool, error) {
+func (s *sender) Configure(config *notification.Config) (bool, error) {
 	// Get configuration
-	var httpConfig WebhookNotifierConfiguration
+	var httpConfig Config
 	if config == nil {
 		return false, nil
 	}
@@ -81,11 +79,11 @@ func (h *WebhookNotifier) Configure(config *config.NotifierConfig) (bool, error)
 	if _, err := url.ParseRequestURI(httpConfig.Endpoint); err != nil {
 		return false, fmt.Errorf("could not parse endpoint URL: %s\n", err)
 	}
-	h.endpoint = httpConfig.Endpoint
+	s.endpoint = httpConfig.Endpoint
 
 	// Setup HTTP client.
 	transport := &http.Transport{}
-	h.client = &http.Client{
+	s.client = &http.Client{
 		Transport: transport,
 		Timeout:   timeout,
 	}
@@ -114,7 +112,7 @@ type notificationEnvelope struct {
 	}
 }
 
-func (h *WebhookNotifier) Send(notification database.VulnerabilityNotification) error {
+func (s *sender) Send(notification database.VulnerabilityNotification) error {
 	// Marshal notification.
 	jsonNotification, err := json.Marshal(notificationEnvelope{struct{ Name string }{notification.Name}})
 	if err != nil {
@@ -122,7 +120,7 @@ func (h *WebhookNotifier) Send(notification database.VulnerabilityNotification)
 	}
 
 	// Send notification via HTTP POST.
-	resp, err := h.client.Post(h.endpoint, "application/json", bytes.NewBuffer(jsonNotification))
+	resp, err := s.client.Post(s.endpoint, "application/json", bytes.NewBuffer(jsonNotification))
 	if err != nil || resp == nil || (resp.StatusCode != 200 && resp.StatusCode != 201) {
 		if resp != nil {
 			return fmt.Errorf("got status %d, expected 200/201", resp.StatusCode)
@@ -134,11 +132,11 @@ func (h *WebhookNotifier) Send(notification database.VulnerabilityNotification)
 	return nil
 }
 
-// loadTLSClientConfig initializes a *tls.Config using the given WebhookNotifierConfiguration.
+// loadTLSClientConfig initializes a *tls.Config using the given Config.
 //
 // If no certificates are given, (nil, nil) is returned.
 // The CA certificate is optional and falls back to the system default.
-func loadTLSClientConfig(cfg *WebhookNotifierConfiguration) (*tls.Config, error) {
+func loadTLSClientConfig(cfg *Config) (*tls.Config, error) {
 	if cfg.CertFile == "" || cfg.KeyFile == "" {
 		return nil, nil
 	}

ext/versionfmt/dpkg/parser.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,6 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package dpkg implements a versionfmt.Parser for version numbers used in dpkg
+// based software packages.
 package dpkg
 
 import (

ext/versionfmt/driver.go

@@ -63,18 +63,20 @@ type Parser interface {
 // if the provided Parser is nil, this function panics.
 func RegisterParser(name string, p Parser) {
 	if name == "" {
-		panic("Could not register a Parser with an empty name")
+		panic("versionfmt: could not register a Parser with an empty name")
 	}
+
 	if p == nil {
-		panic("Could not register a nil Parser")
+		panic("versionfmt: could not register a nil Parser")
 	}
 
 	parsersM.Lock()
 	defer parsersM.Unlock()
 
-	if _, alreadyExists := parsers[name]; alreadyExists {
+	if _, dup := parsers[name]; dup {
-		panic("Parser '" + name + "' is already registered")
+		panic("versionfmt: RegisterParser called twice for " + name)
 	}
+
 	parsers[name] = p
 }
 

ext/versionfmt/rpm/parser.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,6 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package rpm implements a versionfmt.Parser for version numbers used in rpm
+// based software packages.
 package rpm
 
 import (

ext/vulnmdsrc/driver.go

@@ -0,0 +1,88 @@
+// Copyright 2017 clair authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package vulnmdsrc exposes functions to dynamically register vulnerability
+// metadata sources used to update a Clair database.
+package vulnmdsrc
+
+import (
+	"sync"
+
+	"github.com/coreos/clair/database"
+)
+
+var (
+	appendersM sync.RWMutex
+	appenders  = make(map[string]Appender)
+)
+
+// AppendFunc is the type of a callback provided to an Appender.
+type AppendFunc func(metadataKey string, metadata interface{}, severity database.Severity)
+
+// Appender represents anything that can fetch vulnerability metadata and
+// append it to a Vulnerability.
+type Appender interface {
+	// BuildCache loads metadata into memory such that it can be quickly accessed
+	// for future calls to Append.
+	BuildCache(database.Datastore) error
+
+	// AddMetadata adds metadata to the given database.Vulnerability.
+	// It is expected that the fetcher uses .Lock.Lock() when manipulating the Metadata map.
+	// Append
+	Append(vulnName string, callback AppendFunc) error
+
+	// PurgeCache deallocates metadata from memory after all calls to Append are
+	// finished.
+	PurgeCache()
+
+	// Clean deletes any allocated resources.
+	// It is invoked when Clair stops.
+	Clean()
+}
+
+// RegisterAppender makes an Appender available by the provided name.
+//
+// If called twice with the same name, the name is blank, or if the provided
+// Appender is nil, this function panics.
+func RegisterAppender(name string, a Appender) {
+	if name == "" {
+		panic("vulnmdsrc: could not register an Appender with an empty name")
+	}
+
+	if a == nil {
+		panic("vulnmdsrc: could not register a nil Appender")
+	}
+
+	appendersM.Lock()
+	defer appendersM.Unlock()
+
+	if _, dup := appenders[name]; dup {
+		panic("vulnmdsrc: RegisterAppender called twice for " + name)
+	}
+
+	appenders[name] = a
+}
+
+// Appenders returns the list of the registered Appenders.
+func Appenders() map[string]Appender {
+	appendersM.RLock()
+	defer appendersM.RUnlock()
+
+	ret := make(map[string]Appender)
+	for k, v := range appenders {
+		ret[k] = v
+	}
+
+	return ret
+}

ext/vulnmdsrc/nvd/nested_read_closer.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,28 +12,22 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-// Package utils simply defines utility functions and types.
+package nvd
-package utils
 
-import (
+import "io"
-	"bytes"
-	"os/exec"
-)
 
-// Exec runs the given binary with arguments
+// NestedReadCloser wraps an io.Reader and implements io.ReadCloser by closing every embed
-func Exec(dir string, bin string, args ...string) ([]byte, error) {
+// io.ReadCloser.
-	_, err := exec.LookPath(bin)
+// It allows chaining io.ReadCloser together and still keep the ability to close them all in a
-	if err != nil {
+// simple manner.
-		return nil, err
+type NestedReadCloser struct {
-	}
+	io.Reader
-
+	NestedReadClosers []io.ReadCloser
-	cmd := exec.Command(bin, args...)
+}
-	cmd.Dir = dir
+
-
+// Close closes the gzip.Reader and the underlying io.ReadCloser.
-	var buf bytes.Buffer
+func (nrc *NestedReadCloser) Close() {
-	cmd.Stdout = &buf
+	for _, nestedReadCloser := range nrc.NestedReadClosers {
-	cmd.Stderr = &buf
+		nestedReadCloser.Close()
-
+	}
-	err = cmd.Run()
-	return buf.Bytes(), err
 }

ext/vulnmdsrc/nvd/nvd.go

@@ -1,3 +1,19 @@
+// Copyright 2017 clair authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package nvd implements a vulnerability metadata appender using the NIST NVD
+// database.
 package nvd
 
 import (
@@ -12,33 +28,28 @@ import (
 	"os"
 	"strconv"
 	"strings"
-	"sync"
 	"time"
 
-	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/updater"
-	cerrors "github.com/coreos/clair/utils/errors"
-	"github.com/coreos/clair/utils/types"
 	"github.com/coreos/pkg/capnslog"
+
+	"github.com/coreos/clair/database"
+	"github.com/coreos/clair/ext/vulnmdsrc"
+	"github.com/coreos/clair/pkg/commonerr"
 )
 
 const (
 	dataFeedURL     string = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-%s.xml.gz"
 	dataFeedMetaURL string = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-%s.meta"
 
-	metadataKey string = "NVD"
+	appenderName string = "NVD"
 )
 
-var (
+var log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/vulnmdsrc/nvd")
-	log = capnslog.NewPackageLogger("github.com/coreos/clair", "updater/fetchers/metadata_fetchers")
-)
 
-type NVDMetadataFetcher struct {
+type appender struct {
 	localPath      string
 	dataFeedHashes map[string]string
-	lock           sync.Mutex
+	metadata       map[string]NVDMetadata
-
-	metadata map[string]NVDMetadata
 }
 
 type NVDMetadata struct {
@@ -51,46 +62,43 @@ type NVDmetadataCVSSv2 struct {
 }
 
 func init() {
-	updater.RegisterMetadataFetcher("NVD", &NVDMetadataFetcher{})
+	vulnmdsrc.RegisterAppender(appenderName, &appender{})
 }
 
-func (fetcher *NVDMetadataFetcher) Load(datastore database.Datastore) error {
+func (a *appender) BuildCache(datastore database.Datastore) error {
-	fetcher.lock.Lock()
-	defer fetcher.lock.Unlock()
-
 	var err error
-	fetcher.metadata = make(map[string]NVDMetadata)
+	a.metadata = make(map[string]NVDMetadata)
 
 	// Init if necessary.
-	if fetcher.localPath == "" {
+	if a.localPath == "" {
 		// Create a temporary folder to store the NVD data and create hashes struct.
-		if fetcher.localPath, err = ioutil.TempDir(os.TempDir(), "nvd-data"); err != nil {
+		if a.localPath, err = ioutil.TempDir(os.TempDir(), "nvd-data"); err != nil {
-			return cerrors.ErrFilesystem
+			return commonerr.ErrFilesystem
 		}
 
-		fetcher.dataFeedHashes = make(map[string]string)
+		a.dataFeedHashes = make(map[string]string)
 	}
 
 	// Get data feeds.
-	dataFeedReaders, dataFeedHashes, err := getDataFeeds(fetcher.dataFeedHashes, fetcher.localPath)
+	dataFeedReaders, dataFeedHashes, err := getDataFeeds(a.dataFeedHashes, a.localPath)
 	if err != nil {
 		return err
 	}
-	fetcher.dataFeedHashes = dataFeedHashes
+	a.dataFeedHashes = dataFeedHashes
 
 	// Parse data feeds.
 	for dataFeedName, dataFeedReader := range dataFeedReaders {
 		var nvd nvd
 		if err = xml.NewDecoder(dataFeedReader).Decode(&nvd); err != nil {
 			log.Errorf("could not decode NVD data feed '%s': %s", dataFeedName, err)
-			return cerrors.ErrCouldNotParse
+			return commonerr.ErrCouldNotParse
 		}
 
 		// For each entry of this data feed:
 		for _, nvdEntry := range nvd.Entries {
 			// Create metadata entry.
 			if metadata := nvdEntry.Metadata(); metadata != nil {
-				fetcher.metadata[nvdEntry.Name] = *metadata
+				a.metadata[nvdEntry.Name] = *metadata
 			}
 		}
 
@@ -100,42 +108,20 @@ func (fetcher *NVDMetadataFetcher) Load(datastore database.Datastore) error {
 	return nil
 }
 
-func (fetcher *NVDMetadataFetcher) AddMetadata(vulnerability *updater.VulnerabilityWithLock) error {
+func (a *appender) Append(vulnName string, appendFunc vulnmdsrc.AppendFunc) error {
-	fetcher.lock.Lock()
+	if nvdMetadata, ok := a.metadata[vulnName]; ok {
-	defer fetcher.lock.Unlock()
+		appendFunc(appenderName, nvdMetadata, SeverityFromCVSS(nvdMetadata.CVSSv2.Score))
-
-	if nvdMetadata, ok := fetcher.metadata[vulnerability.Name]; ok {
-		vulnerability.Lock.Lock()
-
-		// Create Metadata map if necessary and assign the NVD metadata.
-		if vulnerability.Metadata == nil {
-			vulnerability.Metadata = make(map[string]interface{})
-		}
-		vulnerability.Metadata[metadataKey] = nvdMetadata
-
-		// Set the Severity using the CVSSv2 Score if none is set yet.
-		if vulnerability.Severity == "" || vulnerability.Severity == types.Unknown {
-			vulnerability.Severity = scoreToPriority(nvdMetadata.CVSSv2.Score)
-		}
-
-		vulnerability.Lock.Unlock()
 	}
 
 	return nil
 }
 
-func (fetcher *NVDMetadataFetcher) Unload() {
+func (a *appender) PurgeCache() {
-	fetcher.lock.Lock()
+	a.metadata = nil
-	defer fetcher.lock.Unlock()
-
-	fetcher.metadata = nil
 }
 
-func (fetcher *NVDMetadataFetcher) Clean() {
+func (a *appender) Clean() {
-	fetcher.lock.Lock()
+	os.RemoveAll(a.localPath)
-	defer fetcher.lock.Unlock()
-
-	os.RemoveAll(fetcher.localPath)
 }
 
 func getDataFeeds(dataFeedHashes map[string]string, localPath string) (map[string]NestedReadCloser, map[string]string, error) {
@@ -178,14 +164,14 @@ func getDataFeeds(dataFeedHashes map[string]string, localPath string) (map[strin
 			r, err := http.Get(fmt.Sprintf(dataFeedURL, dataFeedName))
 			if err != nil {
 				log.Errorf("could not download NVD data feed file '%s': %s", dataFeedName, err)
-				return dataFeedReaders, dataFeedHashes, cerrors.ErrCouldNotDownload
+				return dataFeedReaders, dataFeedHashes, commonerr.ErrCouldNotDownload
 			}
 
 			// Un-gzip it.
 			gr, err := gzip.NewReader(r.Body)
 			if err != nil {
 				log.Errorf("could not read NVD data feed file '%s': %s", dataFeedName, err)
-				return dataFeedReaders, dataFeedHashes, cerrors.ErrCouldNotDownload
+				return dataFeedReaders, dataFeedHashes, commonerr.ErrCouldNotDownload
 			}
 
 			// Store it to a file at the same time if possible.
@@ -231,23 +217,25 @@ func getHashFromMetaURL(metaURL string) (string, error) {
 	return "", errors.New("invalid .meta file format")
 }
 
-// scoreToPriority converts the CVSS Score (0.0 - 10.0) into user-friendy
+// SeverityFromCVSS converts the CVSS Score (0.0 - 10.0) into a
-// types.Priority following the qualitative rating scale available in the
+// database.Severity following the qualitative rating scale available in the
 // CVSS v3.0 specification (https://www.first.org/cvss/specification-document),
-// Table 14. The Negligible level is set for CVSS scores between [0, 1),
+// Table 14.
-// replacing the specified None level, originally used for a score of 0.
+//
-func scoreToPriority(score float64) types.Priority {
+// The Negligible level is set for CVSS scores between [0, 1), replacing the
+// specified None level, originally used for a score of 0.
+func SeverityFromCVSS(score float64) database.Severity {
 	switch {
 	case score < 1.0:
-		return types.Negligible
+		return database.NegligibleSeverity
 	case score < 3.9:
-		return types.Low
+		return database.LowSeverity
 	case score < 6.9:
-		return types.Medium
+		return database.MediumSeverity
 	case score < 8.9:
-		return types.High
+		return database.HighSeverity
 	case score <= 10:
-		return types.Critical
+		return database.CriticalSeverity
 	}
-	return types.Unknown
+	return database.UnknownSeverity
 }

ext/vulnmdsrc/nvd/xml.go

@@ -1,3 +1,17 @@
+// Copyright 2017 clair authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 package nvd
 
 import (

ext/vulnsrc/alpine/alpine.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,16 +12,16 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-// Package alpine implements a vulnerability Fetcher using the alpine-secdb
+// Package alpine implements a vulnerability source updater using the
-// git repository.
+// alpine-secdb git repository.
 package alpine
 
 import (
-	"errors"
 	"fmt"
 	"io"
 	"io/ioutil"
 	"os"
+	"os/exec"
 	"strings"
 
 	"gopkg.in/yaml.v2"
@@ -31,10 +31,8 @@ import (
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
-	"github.com/coreos/clair/updater"
+	"github.com/coreos/clair/ext/vulnsrc"
-	"github.com/coreos/clair/utils"
+	"github.com/coreos/clair/pkg/commonerr"
-	cerrors "github.com/coreos/clair/utils/errors"
-	"github.com/coreos/clair/utils/types"
 )
 
 const (
@@ -44,30 +42,22 @@ const (
 	nvdURLPrefix = "https://cve.mitre.org/cgi-bin/cvename.cgi?name="
 )
 
-var (
+var log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/vulnsrc/alpine")
-	// ErrFilesystem is returned when a fetcher fails to interact with the local filesystem.
-	ErrFilesystem = errors.New("updater/fetchers: something went wrong when interacting with the fs")
-
-	// ErrGitFailure is returned when a fetcher fails to interact with git.
-	ErrGitFailure = errors.New("updater/fetchers: something went wrong when interacting with git")
-
-	log = capnslog.NewPackageLogger("github.com/coreos/clair", "updater/fetchers/alpine")
-)
 
 func init() {
-	updater.RegisterFetcher("alpine", &fetcher{})
+	vulnsrc.RegisterUpdater("alpine", &updater{})
 }
 
-type fetcher struct {
+type updater struct {
 	repositoryLocalPath string
 }
 
-func (f *fetcher) FetchUpdate(db database.Datastore) (resp updater.FetcherResponse, err error) {
+func (u *updater) Update(db database.Datastore) (resp vulnsrc.UpdateResponse, err error) {
 	log.Info("fetching Alpine vulnerabilities")
 
 	// Pull the master branch.
 	var commit string
-	commit, err = f.pullRepository()
+	commit, err = u.pullRepository()
 	if err != nil {
 		return
 	}
@@ -90,12 +80,12 @@ func (f *fetcher) FetchUpdate(db database.Datastore) (resp updater.FetcherRespon
 	}
 
 	var namespaces []string
-	namespaces, err = detectNamespaces(f.repositoryLocalPath)
+	namespaces, err = detectNamespaces(u.repositoryLocalPath)
 	// Append any changed vulnerabilities to the response.
 	for _, namespace := range namespaces {
 		var vulns []database.Vulnerability
 		var note string
-		vulns, note, err = parseVulnsFromNamespace(f.repositoryLocalPath, namespace)
+		vulns, note, err = parseVulnsFromNamespace(u.repositoryLocalPath, namespace)
 		if err != nil {
 			return
 		}
@@ -108,6 +98,12 @@ func (f *fetcher) FetchUpdate(db database.Datastore) (resp updater.FetcherRespon
 	return
 }
 
+func (u *updater) Clean() {
+	if u.repositoryLocalPath != "" {
+		os.RemoveAll(u.repositoryLocalPath)
+	}
+}
+
 func detectNamespaces(path string) ([]string, error) {
 	// Open the root directory.
 	dir, err := os.Open(path)
@@ -163,41 +159,40 @@ func parseVulnsFromNamespace(repositoryPath, namespace string) (vulns []database
 	return
 }
 
-func (f *fetcher) pullRepository() (commit string, err error) {
+func (u *updater) pullRepository() (commit string, err error) {
 	// If the repository doesn't exist, clone it.
-	if _, pathExists := os.Stat(f.repositoryLocalPath); f.repositoryLocalPath == "" || os.IsNotExist(pathExists) {
+	if _, pathExists := os.Stat(u.repositoryLocalPath); u.repositoryLocalPath == "" || os.IsNotExist(pathExists) {
-		if f.repositoryLocalPath, err = ioutil.TempDir(os.TempDir(), "alpine-secdb"); err != nil {
+		if u.repositoryLocalPath, err = ioutil.TempDir(os.TempDir(), "alpine-secdb"); err != nil {
-			return "", ErrFilesystem
+			return "", vulnsrc.ErrFilesystem
 		}
 
-		if out, err := utils.Exec(f.repositoryLocalPath, "git", "clone", secdbGitURL, "."); err != nil {
+		cmd := exec.Command("git", "clone", secdbGitURL, ".")
-			f.Clean()
+		cmd.Dir = u.repositoryLocalPath
+		if out, err := cmd.CombinedOutput(); err != nil {
+			u.Clean()
 			log.Errorf("could not pull alpine-secdb repository: %s. output: %s", err, out)
-			return "", cerrors.ErrCouldNotDownload
+			return "", commonerr.ErrCouldNotDownload
 		}
 	} else {
-		// The repository exists and it needs to be refreshed via a pull.
+		// The repository already exists and it needs to be refreshed via a pull.
-		_, err := utils.Exec(f.repositoryLocalPath, "git", "pull")
+		cmd := exec.Command("git", "pull")
-		if err != nil {
+		cmd.Dir = u.repositoryLocalPath
-			return "", ErrGitFailure
+		if _, err := cmd.CombinedOutput(); err != nil {
+			return "", vulnsrc.ErrGitFailure
 		}
 	}
 
-	out, err := utils.Exec(f.repositoryLocalPath, "git", "rev-parse", "HEAD")
+	cmd := exec.Command("git", "rev-parse", "HEAD")
+	cmd.Dir = u.repositoryLocalPath
+	out, err := cmd.CombinedOutput()
 	if err != nil {
-		return "", ErrGitFailure
+		return "", vulnsrc.ErrGitFailure
 	}
 
 	commit = strings.TrimSpace(string(out))
 	return
 }
 
-func (f *fetcher) Clean() {
-	if f.repositoryLocalPath != "" {
-		os.RemoveAll(f.repositoryLocalPath)
-	}
-}
-
 type secdb33File struct {
 	Distro   string `yaml:"distroversion"`
 	Packages []struct {
@@ -232,7 +227,7 @@ func parse33YAML(r io.Reader) (vulns []database.Vulnerability, err error) {
 
 			vulns = append(vulns, database.Vulnerability{
 				Name:     fix,
-				Severity: types.Unknown,
+				Severity: database.UnknownSeverity,
 				Link:     nvdURLPrefix + fix,
 				FixedIn: []database.FeatureVersion{
 					{
@@ -286,7 +281,7 @@ func parse34YAML(r io.Reader) (vulns []database.Vulnerability, err error) {
 
 			for _, vulnStr := range vulnStrs {
 				var vuln database.Vulnerability
-				vuln.Severity = types.Unknown
+				vuln.Severity = database.UnknownSeverity
 				vuln.Name = vulnStr
 				vuln.Link = nvdURLPrefix + vulnStr
 				vuln.FixedIn = []database.FeatureVersion{

ext/vulnsrc/debian/debian.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,6 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package debian implements a vulnerability source updater using the Debian
+// Security Tracker.
 package debian
 
 import (
@@ -28,9 +30,8 @@ import (
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
-	"github.com/coreos/clair/updater"
+	"github.com/coreos/clair/ext/vulnsrc"
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/commonerr"
-	"github.com/coreos/clair/utils/types"
 )
 
 const (
@@ -39,7 +40,7 @@ const (
 	updaterFlag  = "debianUpdater"
 )
 
-var log = capnslog.NewPackageLogger("github.com/coreos/clair", "updater/fetchers/debian")
+var log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/vulnsrc/debian")
 
 type jsonData map[string]map[string]jsonVuln
 
@@ -54,23 +55,20 @@ type jsonRel struct {
 	Urgency      string `json:"urgency"`
 }
 
-// DebianFetcher implements updater.Fetcher for the Debian Security Tracker
+type updater struct{}
-// (https://security-tracker.debian.org).
-type DebianFetcher struct{}
 
 func init() {
-	updater.RegisterFetcher("debian", &DebianFetcher{})
+	vulnsrc.RegisterUpdater("debian", &updater{})
 }
 
-// FetchUpdate fetches vulnerability updates from the Debian Security Tracker.
+func (u *updater) Update(datastore database.Datastore) (resp vulnsrc.UpdateResponse, err error) {
-func (fetcher *DebianFetcher) FetchUpdate(datastore database.Datastore) (resp updater.FetcherResponse, err error) {
 	log.Info("fetching Debian vulnerabilities")
 
 	// Download JSON.
 	r, err := http.Get(url)
 	if err != nil {
 		log.Errorf("could not download Debian's update: %s", err)
-		return resp, cerrors.ErrCouldNotDownload
+		return resp, commonerr.ErrCouldNotDownload
 	}
 
 	// Get the SHA-1 of the latest update's JSON data
@@ -88,7 +86,9 @@ func (fetcher *DebianFetcher) FetchUpdate(datastore database.Datastore) (resp up
 	return resp, nil
 }
 
-func buildResponse(jsonReader io.Reader, latestKnownHash string) (resp updater.FetcherResponse, err error) {
+func (u *updater) Clean() {}
+
+func buildResponse(jsonReader io.Reader, latestKnownHash string) (resp vulnsrc.UpdateResponse, err error) {
 	hash := latestKnownHash
 
 	// Defer the addition of flag information to the response.
@@ -109,7 +109,7 @@ func buildResponse(jsonReader io.Reader, latestKnownHash string) (resp updater.F
 	err = json.NewDecoder(teedJSONReader).Decode(&data)
 	if err != nil {
 		log.Errorf("could not unmarshal Debian's JSON: %s", err)
-		return resp, cerrors.ErrCouldNotParse
+		return resp, commonerr.ErrCouldNotParse
 	}
 
 	// Calculate the hash and skip updating if the hash has been seen before.
@@ -157,17 +157,17 @@ func parseDebianJSON(data *jsonData) (vulnerabilities []database.Vulnerability,
 					vulnerability = &database.Vulnerability{
 						Name:        vulnName,
 						Link:        strings.Join([]string{cveURLPrefix, "/", vulnName}, ""),
-						Severity:    types.Unknown,
+						Severity:    database.UnknownSeverity,
 						Description: vulnNode.Description,
 					}
 				}
 
 				// Set the priority of the vulnerability.
 				// In the JSON, a vulnerability has one urgency per package it affects.
-				// The highest urgency should be the one set.
+				severity := SeverityFromUrgency(releaseNode.Urgency)
-				urgency := urgencyToSeverity(releaseNode.Urgency)
+				if severity.Compare(vulnerability.Severity) > 0 {
-				if urgency.Compare(vulnerability.Severity) > 0 {
+					// The highest urgency should be the one set.
-					vulnerability.Severity = urgency
+					vulnerability.Severity = severity
 				}
 
 				// Determine the version of the package the vulnerability affects.
@@ -218,42 +218,41 @@ func parseDebianJSON(data *jsonData) (vulnerabilities []database.Vulnerability,
 	return
 }
 
-func urgencyToSeverity(urgency string) types.Priority {
+// SeverityFromUrgency converts the urgency scale used by the Debian Security
+// Bug Tracker into a database.Severity.
+func SeverityFromUrgency(urgency string) database.Severity {
 	switch urgency {
 	case "not yet assigned":
-		return types.Unknown
+		return database.UnknownSeverity
 
 	case "end-of-life":
 		fallthrough
 	case "unimportant":
-		return types.Negligible
+		return database.NegligibleSeverity
 
 	case "low":
 		fallthrough
 	case "low*":
 		fallthrough
 	case "low**":
-		return types.Low
+		return database.LowSeverity
 
 	case "medium":
 		fallthrough
 	case "medium*":
 		fallthrough
 	case "medium**":
-		return types.Medium
+		return database.MediumSeverity
 
 	case "high":
 		fallthrough
 	case "high*":
 		fallthrough
 	case "high**":
-		return types.High
+		return database.HighSeverity
 
 	default:
-		log.Warningf("could not determine vulnerability priority from: %s", urgency)
+		log.Warningf("could not determine vulnerability severity from: %s", urgency)
-		return types.Unknown
+		return database.UnknownSeverity
 	}
 }
-
-// Clean deletes any allocated resources.
-func (fetcher *DebianFetcher) Clean() {}

ext/vulnsrc/debian/debian_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -23,7 +23,6 @@ import (
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
-	"github.com/coreos/clair/utils/types"
 	"github.com/stretchr/testify/assert"
 )
 
@@ -37,7 +36,7 @@ func TestDebianParser(t *testing.T) {
 		for _, vulnerability := range response.Vulnerabilities {
 			if vulnerability.Name == "CVE-2015-1323" {
 				assert.Equal(t, "https://security-tracker.debian.org/tracker/CVE-2015-1323", vulnerability.Link)
-				assert.Equal(t, types.Low, vulnerability.Severity)
+				assert.Equal(t, database.LowSeverity, vulnerability.Severity)
 				assert.Equal(t, "This vulnerability is not very dangerous.", vulnerability.Description)
 
 				expectedFeatureVersions := []database.FeatureVersion{
@@ -68,7 +67,7 @@ func TestDebianParser(t *testing.T) {
 				}
 			} else if vulnerability.Name == "CVE-2003-0779" {
 				assert.Equal(t, "https://security-tracker.debian.org/tracker/CVE-2003-0779", vulnerability.Link)
-				assert.Equal(t, types.High, vulnerability.Severity)
+				assert.Equal(t, database.HighSeverity, vulnerability.Severity)
 				assert.Equal(t, "But this one is very dangerous.", vulnerability.Description)
 
 				expectedFeatureVersions := []database.FeatureVersion{
@@ -109,7 +108,7 @@ func TestDebianParser(t *testing.T) {
 				}
 			} else if vulnerability.Name == "CVE-2013-2685" {
 				assert.Equal(t, "https://security-tracker.debian.org/tracker/CVE-2013-2685", vulnerability.Link)
-				assert.Equal(t, types.Negligible, vulnerability.Severity)
+				assert.Equal(t, database.NegligibleSeverity, vulnerability.Severity)
 				assert.Equal(t, "Un-affected packages.", vulnerability.Description)
 
 				expectedFeatureVersions := []database.FeatureVersion{

ext/vulnsrc/driver.go

@@ -0,0 +1,90 @@
+// Copyright 2017 clair authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package vulnsrc exposes functions to dynamically register vulnerability
+// sources used to update a Clair database.
+package vulnsrc
+
+import (
+	"errors"
+	"sync"
+
+	"github.com/coreos/clair/database"
+)
+
+var (
+	// ErrFilesystem is returned when a fetcher fails to interact with the local filesystem.
+	ErrFilesystem = errors.New("vulnsrc: something went wrong when interacting with the fs")
+
+	// ErrGitFailure is returned when a fetcher fails to interact with git.
+	ErrGitFailure = errors.New("vulnsrc: something went wrong when interacting with git")
+
+	updatersM sync.RWMutex
+	updaters  = make(map[string]Updater)
+)
+
+// UpdateResponse represents the sum of results of an update.
+type UpdateResponse struct {
+	FlagName        string
+	FlagValue       string
+	Notes           []string
+	Vulnerabilities []database.Vulnerability
+}
+
+// Updater represents anything that can fetch vulnerabilities and insert them
+// into a Clair datastore.
+type Updater interface {
+	// Update gets vulnerability updates.
+	Update(database.Datastore) (UpdateResponse, error)
+
+	// Clean deletes any allocated resources.
+	// It is invoked when Clair stops.
+	Clean()
+}
+
+// RegisterUpdater makes an Updater available by the provided name.
+//
+// If called twice with the same name, the name is blank, or if the provided
+// Updater is nil, this function panics.
+func RegisterUpdater(name string, u Updater) {
+	if name == "" {
+		panic("vulnsrc: could not register an Updater with an empty name")
+	}
+
+	if u == nil {
+		panic("vulnsrc: could not register a nil Updater")
+	}
+
+	updatersM.Lock()
+	defer updatersM.Unlock()
+
+	if _, dup := updaters[name]; dup {
+		panic("vulnsrc: RegisterUpdater called twice for " + name)
+	}
+
+	updaters[name] = u
+}
+
+// Updaters returns the list of the registered Updaters.
+func Updaters() map[string]Updater {
+	updatersM.RLock()
+	defer updatersM.RUnlock()
+
+	ret := make(map[string]Updater)
+	for k, v := range updaters {
+		ret[k] = v
+	}
+
+	return ret
+}

ext/vulnsrc/oracle/oracle.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,6 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package oracle implements a vulnerability source updater using the
+// Oracle Linux OVAL Database.
 package oracle
 
 import (
@@ -23,13 +25,13 @@ import (
 	"strconv"
 	"strings"
 
+	"github.com/coreos/pkg/capnslog"
+
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt"
 	"github.com/coreos/clair/ext/versionfmt/rpm"
-	"github.com/coreos/clair/updater"
+	"github.com/coreos/clair/ext/vulnsrc"
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/commonerr"
-	"github.com/coreos/clair/utils/types"
-	"github.com/coreos/pkg/capnslog"
 )
 
 const (
@@ -47,7 +49,7 @@ var (
 
 	elsaRegexp = regexp.MustCompile(`com.oracle.elsa-(\d+).xml`)
 
-	log = capnslog.NewPackageLogger("github.com/coreos/clair", "updater/fetchers/oracle")
+	log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/vulnsrc/oracle")
 )
 
 type oval struct {
@@ -77,16 +79,13 @@ type criterion struct {
 	Comment string `xml:"comment,attr"`
 }
 
-// OracleFetcher implements updater.Fetcher and gets vulnerability updates from
+type updater struct{}
-// the Oracle Linux OVAL definitions.
-type OracleFetcher struct{}
 
 func init() {
-	updater.RegisterFetcher("Oracle", &OracleFetcher{})
+	vulnsrc.RegisterUpdater("oracle", &updater{})
 }
 
-// FetchUpdate gets vulnerability updates from the Oracle Linux OVAL definitions.
+func (u *updater) Update(datastore database.Datastore) (resp vulnsrc.UpdateResponse, err error) {
-func (f *OracleFetcher) FetchUpdate(datastore database.Datastore) (resp updater.FetcherResponse, err error) {
 	log.Info("fetching Oracle Linux vulnerabilities")
 
 	// Get the first ELSA we have to manage.
@@ -104,7 +103,7 @@ func (f *OracleFetcher) FetchUpdate(datastore database.Datastore) (resp updater.
 	r, err := http.Get(ovalURI)
 	if err != nil {
 		log.Errorf("could not download Oracle's update list: %s", err)
-		return resp, cerrors.ErrCouldNotDownload
+		return resp, commonerr.ErrCouldNotDownload
 	}
 	defer r.Body.Close()
 
@@ -127,7 +126,7 @@ func (f *OracleFetcher) FetchUpdate(datastore database.Datastore) (resp updater.
 		r, err := http.Get(ovalURI + elsaFilePrefix + strconv.Itoa(elsa) + ".xml")
 		if err != nil {
 			log.Errorf("could not download Oracle's update file: %s", err)
-			return resp, cerrors.ErrCouldNotDownload
+			return resp, commonerr.ErrCouldNotDownload
 		}
 
 		// Parse the XML.
@@ -153,13 +152,15 @@ func (f *OracleFetcher) FetchUpdate(datastore database.Datastore) (resp updater.
 	return resp, nil
 }
 
+func (u *updater) Clean() {}
+
 func parseELSA(ovalReader io.Reader) (vulnerabilities []database.Vulnerability, err error) {
 	// Decode the XML.
 	var ov oval
 	err = xml.NewDecoder(ovalReader).Decode(&ov)
 	if err != nil {
 		log.Errorf("could not decode Oracle's XML: %s", err)
-		err = cerrors.ErrCouldNotParse
+		err = commonerr.ErrCouldNotParse
 		return
 	}
 
@@ -171,7 +172,7 @@ func parseELSA(ovalReader io.Reader) (vulnerabilities []database.Vulnerability,
 			vulnerability := database.Vulnerability{
 				Name:        name(definition),
 				Link:        link(definition),
-				Severity:    priority(definition),
+				Severity:    severity(definition),
 				Description: description(definition),
 			}
 			for _, p := range pkgs {
@@ -335,27 +336,20 @@ func link(def definition) (link string) {
 	return
 }
 
-func priority(def definition) types.Priority {
+func severity(def definition) database.Severity {
-	// Parse the priority.
+	switch strings.ToLower(def.Severity) {
-	priority := strings.ToLower(def.Severity)
-
-	// Normalize the priority.
-	switch priority {
 	case "n/a":
-		return types.Negligible
+		return database.NegligibleSeverity
 	case "low":
-		return types.Low
+		return database.LowSeverity
 	case "moderate":
-		return types.Medium
+		return database.MediumSeverity
 	case "important":
-		return types.High
+		return database.HighSeverity
 	case "critical":
-		return types.Critical
+		return database.CriticalSeverity
 	default:
-		log.Warningf("could not determine vulnerability priority from: %s.", priority)
+		log.Warningf("could not determine vulnerability severity from: %s.", def.Severity)
-		return types.Unknown
+		return database.UnknownSeverity
 	}
 }
-
-// Clean deletes any allocated resources.
-func (f *OracleFetcher) Clean() {}

ext/vulnsrc/oracle/oracle_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -22,7 +22,6 @@ import (
 
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt/rpm"
-	"github.com/coreos/clair/utils/types"
 	"github.com/stretchr/testify/assert"
 )
 
@@ -38,7 +37,7 @@ func TestOracleParser(t *testing.T) {
 	if assert.Nil(t, err) && assert.Len(t, vulnerabilities, 1) {
 		assert.Equal(t, "ELSA-2015-1193", vulnerabilities[0].Name)
 		assert.Equal(t, "http://linux.oracle.com/errata/ELSA-2015-1193.html", vulnerabilities[0].Link)
-		assert.Equal(t, types.Medium, vulnerabilities[0].Severity)
+		assert.Equal(t, database.MediumSeverity, vulnerabilities[0].Severity)
 		assert.Equal(t, ` [3.1.1-7] Resolves: rhbz#1217104 CVE-2015-0252 `, vulnerabilities[0].Description)
 
 		expectedFeatureVersions := []database.FeatureVersion{
@@ -86,7 +85,7 @@ func TestOracleParser(t *testing.T) {
 	if assert.Nil(t, err) && assert.Len(t, vulnerabilities, 1) {
 		assert.Equal(t, "ELSA-2015-1207", vulnerabilities[0].Name)
 		assert.Equal(t, "http://linux.oracle.com/errata/ELSA-2015-1207.html", vulnerabilities[0].Link)
-		assert.Equal(t, types.Critical, vulnerabilities[0].Severity)
+		assert.Equal(t, database.CriticalSeverity, vulnerabilities[0].Severity)
 		assert.Equal(t, ` [38.1.0-1.0.1.el7_1] - Add firefox-oracle-default-prefs.js and remove the corresponding Red Hat file [38.1.0-1] - Update to 38.1.0 ESR [38.0.1-2] - Fixed rhbz#1222807 by removing preun section `, vulnerabilities[0].Description)
 		expectedFeatureVersions := []database.FeatureVersion{
 			{

ext/vulnsrc/rhel/rhel.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,6 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package rhel implements a vulnerability source updater using the
+// Red Hat Linux OVAL Database.
 package rhel
 
 import (
@@ -28,9 +30,8 @@ import (
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt"
 	"github.com/coreos/clair/ext/versionfmt/rpm"
-	"github.com/coreos/clair/updater"
+	"github.com/coreos/clair/ext/vulnsrc"
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/commonerr"
-	"github.com/coreos/clair/utils/types"
 )
 
 const (
@@ -82,17 +83,14 @@ type criterion struct {
 	Comment string `xml:"comment,attr"`
 }
 
-// RHELFetcher implements updater.Fetcher and gets vulnerability updates from
+type updater struct{}
-// the Red Hat OVAL definitions.
-type RHELFetcher struct{}
 
 func init() {
-	updater.RegisterFetcher("Red Hat", &RHELFetcher{})
+	vulnsrc.RegisterUpdater("rhel", &updater{})
 }
 
-// FetchUpdate gets vulnerability updates from the Red Hat OVAL definitions.
+func (u *updater) Update(datastore database.Datastore) (resp vulnsrc.UpdateResponse, err error) {
-func (f *RHELFetcher) FetchUpdate(datastore database.Datastore) (resp updater.FetcherResponse, err error) {
+	log.Info("fetching RHEL vulnerabilities")
-	log.Info("fetching Red Hat vulnerabilities")
 
 	// Get the first RHSA we have to manage.
 	flagValue, err := datastore.GetKeyValue(updaterFlag)
@@ -108,7 +106,7 @@ func (f *RHELFetcher) FetchUpdate(datastore database.Datastore) (resp updater.Fe
 	r, err := http.Get(ovalURI)
 	if err != nil {
 		log.Errorf("could not download RHEL's update list: %s", err)
-		return resp, cerrors.ErrCouldNotDownload
+		return resp, commonerr.ErrCouldNotDownload
 	}
 
 	// Get the list of RHSAs that we have to process.
@@ -130,7 +128,7 @@ func (f *RHELFetcher) FetchUpdate(datastore database.Datastore) (resp updater.Fe
 		r, err := http.Get(ovalURI + rhsaFilePrefix + strconv.Itoa(rhsa) + ".xml")
 		if err != nil {
 			log.Errorf("could not download RHEL's update file: %s", err)
-			return resp, cerrors.ErrCouldNotDownload
+			return resp, commonerr.ErrCouldNotDownload
 		}
 
 		// Parse the XML.
@@ -156,13 +154,15 @@ func (f *RHELFetcher) FetchUpdate(datastore database.Datastore) (resp updater.Fe
 	return resp, nil
 }
 
+func (u *updater) Clean() {}
+
 func parseRHSA(ovalReader io.Reader) (vulnerabilities []database.Vulnerability, err error) {
 	// Decode the XML.
 	var ov oval
 	err = xml.NewDecoder(ovalReader).Decode(&ov)
 	if err != nil {
 		log.Errorf("could not decode RHEL's XML: %s", err)
-		err = cerrors.ErrCouldNotParse
+		err = commonerr.ErrCouldNotParse
 		return
 	}
 
@@ -174,7 +174,7 @@ func parseRHSA(ovalReader io.Reader) (vulnerabilities []database.Vulnerability,
 			vulnerability := database.Vulnerability{
 				Name:        name(definition),
 				Link:        link(definition),
-				Severity:    priority(definition),
+				Severity:    severity(definition),
 				Description: description(definition),
 			}
 			for _, p := range pkgs {
@@ -343,25 +343,18 @@ func link(def definition) (link string) {
 	return
 }
 
-func priority(def definition) types.Priority {
+func severity(def definition) database.Severity {
-	// Parse the priority.
+	switch strings.TrimSpace(def.Title[strings.LastIndex(def.Title, "(")+1 : len(def.Title)-1]) {
-	priority := strings.TrimSpace(def.Title[strings.LastIndex(def.Title, "(")+1 : len(def.Title)-1])
-
-	// Normalize the priority.
-	switch priority {
 	case "Low":
-		return types.Low
+		return database.LowSeverity
 	case "Moderate":
-		return types.Medium
+		return database.MediumSeverity
 	case "Important":
-		return types.High
+		return database.HighSeverity
 	case "Critical":
-		return types.Critical
+		return database.CriticalSeverity
 	default:
-		log.Warning("could not determine vulnerability priority from: %s.", priority)
+		log.Warning("could not determine vulnerability severity from: %s.", def.Title)
-		return types.Unknown
+		return database.UnknownSeverity
 	}
 }
-
-// Clean deletes any allocated resources.
-func (f *RHELFetcher) Clean() {}

ext/vulnsrc/rhel/rhel_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -22,7 +22,6 @@ import (
 
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt/rpm"
-	"github.com/coreos/clair/utils/types"
 	"github.com/stretchr/testify/assert"
 )
 
@@ -36,7 +35,7 @@ func TestRHELParser(t *testing.T) {
 	if assert.Nil(t, err) && assert.Len(t, vulnerabilities, 1) {
 		assert.Equal(t, "RHSA-2015:1193", vulnerabilities[0].Name)
 		assert.Equal(t, "https://rhn.redhat.com/errata/RHSA-2015-1193.html", vulnerabilities[0].Link)
-		assert.Equal(t, types.Medium, vulnerabilities[0].Severity)
+		assert.Equal(t, database.MediumSeverity, vulnerabilities[0].Severity)
 		assert.Equal(t, `Xerces-C is a validating XML parser written in a portable subset of C++. A flaw was found in the way the Xerces-C XML parser processed certain XML documents. A remote attacker could provide specially crafted XML input that, when parsed by an application using Xerces-C, would cause that application to crash.`, vulnerabilities[0].Description)
 
 		expectedFeatureVersions := []database.FeatureVersion{
@@ -83,7 +82,7 @@ func TestRHELParser(t *testing.T) {
 	if assert.Nil(t, err) && assert.Len(t, vulnerabilities, 1) {
 		assert.Equal(t, "RHSA-2015:1207", vulnerabilities[0].Name)
 		assert.Equal(t, "https://rhn.redhat.com/errata/RHSA-2015-1207.html", vulnerabilities[0].Link)
-		assert.Equal(t, types.Critical, vulnerabilities[0].Severity)
+		assert.Equal(t, database.CriticalSeverity, vulnerabilities[0].Severity)
 		assert.Equal(t, `Mozilla Firefox is an open source web browser. XULRunner provides the XUL Runtime environment for Mozilla Firefox. Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.`, vulnerabilities[0].Description)
 
 		expectedFeatureVersions := []database.FeatureVersion{

ext/vulnsrc/ubuntu/ubuntu.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,16 +12,18 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+// Package ubuntu implements a vulnerability source updater using the
+// Ubuntu CVE Tracker.
 package ubuntu
 
 import (
 	"bufio"
 	"bytes"
-	"errors"
 	"fmt"
 	"io"
 	"io/ioutil"
 	"os"
+	"os/exec"
 	"regexp"
 	"strconv"
 	"strings"
@@ -31,10 +33,8 @@ import (
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt"
 	"github.com/coreos/clair/ext/versionfmt/dpkg"
-	"github.com/coreos/clair/updater"
+	"github.com/coreos/clair/ext/vulnsrc"
-	"github.com/coreos/clair/utils"
+	"github.com/coreos/clair/pkg/commonerr"
-	cerrors "github.com/coreos/clair/utils/errors"
-	"github.com/coreos/clair/utils/types"
 )
 
 const (
@@ -75,33 +75,27 @@ var (
 	affectsCaptureRegexp      = regexp.MustCompile(`(?P<release>.*)_(?P<package>.*): (?P<status>[^\s]*)( \(+(?P<note>[^()]*)\)+)?`)
 	affectsCaptureRegexpNames = affectsCaptureRegexp.SubexpNames()
 
-	log = capnslog.NewPackageLogger("github.com/coreos/clair", "updater/fetchers/ubuntu")
+	log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/vulnsrc/ubuntu")
-
-	// ErrFilesystem is returned when a fetcher fails to interact with the local filesystem.
-	ErrFilesystem = errors.New("updater/fetchers: something went wrong when interacting with the fs")
 )
 
-// UbuntuFetcher implements updater.Fetcher and gets vulnerability updates from
+type updater struct {
-// the Ubuntu CVE Tracker.
-type UbuntuFetcher struct {
 	repositoryLocalPath string
 }
 
 func init() {
-	updater.RegisterFetcher("Ubuntu", &UbuntuFetcher{})
+	vulnsrc.RegisterUpdater("ubuntu", &updater{})
 }
 
-// FetchUpdate gets vulnerability updates from the Ubuntu CVE Tracker.
+func (u *updater) Update(datastore database.Datastore) (resp vulnsrc.UpdateResponse, err error) {
-func (fetcher *UbuntuFetcher) FetchUpdate(datastore database.Datastore) (resp updater.FetcherResponse, err error) {
 	log.Info("fetching Ubuntu vulnerabilities")
 
 	// Pull the bzr repository.
-	if err = fetcher.pullRepository(); err != nil {
+	if err = u.pullRepository(); err != nil {
 		return resp, err
 	}
 
 	// Get revision number.
-	revisionNumber, err := getRevisionNumber(fetcher.repositoryLocalPath)
+	revisionNumber, err := getRevisionNumber(u.repositoryLocalPath)
 	if err != nil {
 		return resp, err
 	}
@@ -113,7 +107,7 @@ func (fetcher *UbuntuFetcher) FetchUpdate(datastore database.Datastore) (resp up
 	}
 
 	// Get the list of vulnerabilities that we have to update.
-	modifiedCVE, err := collectModifiedVulnerabilities(revisionNumber, dbRevisionNumber, fetcher.repositoryLocalPath)
+	modifiedCVE, err := collectModifiedVulnerabilities(revisionNumber, dbRevisionNumber, u.repositoryLocalPath)
 	if err != nil {
 		return resp, err
 	}
@@ -121,7 +115,7 @@ func (fetcher *UbuntuFetcher) FetchUpdate(datastore database.Datastore) (resp up
 	notes := make(map[string]struct{})
 	for cvePath := range modifiedCVE {
 		// Open the CVE file.
-		file, err := os.Open(fetcher.repositoryLocalPath + "/" + cvePath)
+		file, err := os.Open(u.repositoryLocalPath + "/" + cvePath)
 		if err != nil {
 			// This can happen when a file is modified and then moved in another
 			// commit.
@@ -166,45 +160,57 @@ func (fetcher *UbuntuFetcher) FetchUpdate(datastore database.Datastore) (resp up
 	return
 }
 
-func (fetcher *UbuntuFetcher) pullRepository() (err error) {
+func (u *updater) Clean() {
+	os.RemoveAll(u.repositoryLocalPath)
+}
+
+func (u *updater) pullRepository() (err error) {
 	// Determine whether we should branch or pull.
-	if _, pathExists := os.Stat(fetcher.repositoryLocalPath); fetcher.repositoryLocalPath == "" || os.IsNotExist(pathExists) {
+	if _, pathExists := os.Stat(u.repositoryLocalPath); u.repositoryLocalPath == "" || os.IsNotExist(pathExists) {
 		// Create a temporary folder to store the repository.
-		if fetcher.repositoryLocalPath, err = ioutil.TempDir(os.TempDir(), "ubuntu-cve-tracker"); err != nil {
+		if u.repositoryLocalPath, err = ioutil.TempDir(os.TempDir(), "ubuntu-cve-tracker"); err != nil {
-			return ErrFilesystem
+			return vulnsrc.ErrFilesystem
 		}
 
 		// Branch repository.
-		if out, err := utils.Exec(fetcher.repositoryLocalPath, "bzr", "branch", "--use-existing-dir", trackerRepository, "."); err != nil {
+		cmd := exec.Command("bzr", "branch", "--use-existing-dir", trackerRepository, ".")
+		cmd.Dir = u.repositoryLocalPath
+		if out, err := cmd.CombinedOutput(); err != nil {
 			log.Errorf("could not branch Ubuntu repository: %s. output: %s", err, out)
-			return cerrors.ErrCouldNotDownload
+			return commonerr.ErrCouldNotDownload
 		}
 
 		return nil
 	}
 
 	// Pull repository.
-	if out, err := utils.Exec(fetcher.repositoryLocalPath, "bzr", "pull", "--overwrite"); err != nil {
+	cmd := exec.Command("bzr", "pull", "--overwrite")
-		os.RemoveAll(fetcher.repositoryLocalPath)
+	cmd.Dir = u.repositoryLocalPath
+	if out, err := cmd.CombinedOutput(); err != nil {
+		os.RemoveAll(u.repositoryLocalPath)
 
 		log.Errorf("could not pull Ubuntu repository: %s. output: %s", err, out)
-		return cerrors.ErrCouldNotDownload
+		return commonerr.ErrCouldNotDownload
 	}
 
 	return nil
 }
 
 func getRevisionNumber(pathToRepo string) (int, error) {
-	out, err := utils.Exec(pathToRepo, "bzr", "revno")
+	cmd := exec.Command("bzr", "revno")
+	cmd.Dir = pathToRepo
+	out, err := cmd.CombinedOutput()
 	if err != nil {
 		log.Errorf("could not get Ubuntu repository's revision number: %s. output: %s", err, out)
-		return 0, cerrors.ErrCouldNotDownload
+		return 0, commonerr.ErrCouldNotDownload
 	}
+
 	revno, err := strconv.Atoi(strings.TrimSpace(string(out)))
 	if err != nil {
 		log.Errorf("could not parse Ubuntu repository's revision number: %s. output: %s", err, out)
-		return 0, cerrors.ErrCouldNotDownload
+		return 0, commonerr.ErrCouldNotDownload
 	}
+
 	return revno, nil
 }
 
@@ -217,14 +223,14 @@ func collectModifiedVulnerabilities(revision int, dbRevision, repositoryLocalPat
 			d, err := os.Open(repositoryLocalPath + "/" + folder)
 			if err != nil {
 				log.Errorf("could not open Ubuntu vulnerabilities repository's folder: %s", err)
-				return nil, ErrFilesystem
+				return nil, vulnsrc.ErrFilesystem
 			}
 
 			// Get the FileInfo of all the files in the directory.
 			names, err := d.Readdirnames(-1)
 			if err != nil {
 				log.Errorf("could not read Ubuntu vulnerabilities repository's folder:: %s.", err)
-				return nil, ErrFilesystem
+				return nil, vulnsrc.ErrFilesystem
 			}
 
 			// Add the vulnerabilities to the list.
@@ -253,10 +259,12 @@ func collectModifiedVulnerabilities(revision int, dbRevision, repositoryLocalPat
 	}
 
 	// Handle a database that needs upgrading.
-	out, err := utils.Exec(repositoryLocalPath, "bzr", "log", "--verbose", "-r"+strconv.Itoa(dbRevisionInt+1)+"..", "-n0")
+	cmd := exec.Command("bzr", "log", "--verbose", "-r"+strconv.Itoa(dbRevisionInt+1)+"..", "-n0")
+	cmd.Dir = repositoryLocalPath
+	out, err := cmd.CombinedOutput()
 	if err != nil {
 		log.Errorf("could not get Ubuntu vulnerabilities repository logs: %s. output: %s", err, out)
-		return nil, cerrors.ErrCouldNotDownload
+		return nil, commonerr.ErrCouldNotDownload
 	}
 
 	scanner := bufio.NewScanner(bytes.NewReader(out))
@@ -302,7 +310,7 @@ func parseUbuntuCVE(fileContent io.Reader) (vulnerability database.Vulnerability
 				priority = priority[:strings.Index(priority, " ")]
 			}
 
-			vulnerability.Severity = ubuntuPriorityToSeverity(priority)
+			vulnerability.Severity = SeverityFromPriority(priority)
 			continue
 		}
 
@@ -389,33 +397,30 @@ func parseUbuntuCVE(fileContent io.Reader) (vulnerability database.Vulnerability
 
 	// If no priority has been provided (CVE-2007-0667 for instance), set the priority to Unknown
 	if vulnerability.Severity == "" {
-		vulnerability.Severity = types.Unknown
+		vulnerability.Severity = database.UnknownSeverity
 	}
 
 	return
 }
 
-func ubuntuPriorityToSeverity(priority string) types.Priority {
+// SeverityFromPriority converts an priority from the Ubuntu CVE Tracker into
+// a database.Severity.
+func SeverityFromPriority(priority string) database.Severity {
 	switch priority {
 	case "untriaged":
-		return types.Unknown
+		return database.UnknownSeverity
 	case "negligible":
-		return types.Negligible
+		return database.NegligibleSeverity
 	case "low":
-		return types.Low
+		return database.LowSeverity
 	case "medium":
-		return types.Medium
+		return database.MediumSeverity
 	case "high":
-		return types.High
+		return database.HighSeverity
 	case "critical":
-		return types.Critical
+		return database.CriticalSeverity
+	default:
+		log.Warning("could not determine a vulnerability severity from: %s", priority)
+		return database.UnknownSeverity
 	}
-
-	log.Warning("Could not determine a vulnerability priority from: %s", priority)
-	return types.Unknown
-}
-
-// Clean deletes any allocated resources.
-func (fetcher *UbuntuFetcher) Clean() {
-	os.RemoveAll(fetcher.repositoryLocalPath)
 }

ext/vulnsrc/ubuntu/ubuntu_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -24,7 +24,6 @@ import (
 
 	"github.com/coreos/clair/database"
 	"github.com/coreos/clair/ext/versionfmt"
-	"github.com/coreos/clair/utils/types"
 )
 
 func TestUbuntuParser(t *testing.T) {
@@ -37,7 +36,7 @@ func TestUbuntuParser(t *testing.T) {
 	vulnerability, unknownReleases, err := parseUbuntuCVE(testData)
 	if assert.Nil(t, err) {
 		assert.Equal(t, "CVE-2015-4471", vulnerability.Name)
-		assert.Equal(t, types.Medium, vulnerability.Severity)
+		assert.Equal(t, database.MediumSeverity, vulnerability.Severity)
 		assert.Equal(t, "Off-by-one error in the lzxd_decompress function in lzxd.c in libmspack before 0.5 allows remote attackers to cause a denial of service (buffer under-read and application crash) via a crafted CAB archive.", vulnerability.Description)
 
 		// Unknown release (line 28)

notifier.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,36 +12,29 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-// Package notifier fetches notifications from the database and informs the specified remote handler
+package clair
-// about their existences, inviting the third party to actively query the API about it.
-package notifier
 
 import (
 	"time"
 
-	"github.com/coreos/pkg/capnslog"
 	"github.com/coreos/pkg/timeutil"
 	"github.com/pborman/uuid"
 	"github.com/prometheus/client_golang/prometheus"
 
-	"github.com/coreos/clair/config"
 	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/utils"
+	"github.com/coreos/clair/ext/notification"
-	cerrors "github.com/coreos/clair/utils/errors"
+	"github.com/coreos/clair/pkg/commonerr"
+	"github.com/coreos/clair/pkg/stopper"
 )
 
 const (
-	checkInterval       = 5 * time.Minute
+	notifierCheckInterval       = 5 * time.Minute
-	refreshLockDuration = time.Minute * 2
+	notifierMaxBackOff          = 15 * time.Minute
-	lockDuration        = time.Minute*8 + refreshLockDuration
+	notifierLockRefreshDuration = time.Minute * 2
-	maxBackOff          = 15 * time.Minute
+	notifierLockDuration        = time.Minute*8 + notifierLockRefreshDuration
 )
 
 var (
-	log = capnslog.NewPackageLogger("github.com/coreos/clair", "notifier")
-
-	notifiers = make(map[string]Notifier)
-
 	promNotifierLatencyMilliseconds = prometheus.NewHistogram(prometheus.HistogramOpts{
 		Name: "clair_notifier_latency_milliseconds",
 		Help: "Time it takes to send a notification after it's been created.",
@@ -53,57 +46,30 @@ var (
 	}, []string{"backend"})
 )
 
-// Notifier represents anything that can transmit notifications.
-type Notifier interface {
-	// Configure attempts to initialize the notifier with the provided configuration.
-	// It returns whether the notifier is enabled or not.
-	Configure(*config.NotifierConfig) (bool, error)
-	// Send informs the existence of the specified notification.
-	Send(notification database.VulnerabilityNotification) error
-}
-
 func init() {
 	prometheus.MustRegister(promNotifierLatencyMilliseconds)
 	prometheus.MustRegister(promNotifierBackendErrorsTotal)
 }
 
-// RegisterNotifier makes a Fetcher available by the provided name.
+// RunNotifier begins a process that checks for new notifications that should
-// If Register is called twice with the same name or if driver is nil,
+// be sent out to third parties.
-// it panics.
+func RunNotifier(config *notification.Config, datastore database.Datastore, stopper *stopper.Stopper) {
-func RegisterNotifier(name string, n Notifier) {
-	if name == "" {
-		panic("notifier: could not register a Notifier with an empty name")
-	}
-
-	if n == nil {
-		panic("notifier: could not register a nil Notifier")
-	}
-
-	if _, dup := notifiers[name]; dup {
-		panic("notifier: RegisterNotifier called twice for " + name)
-	}
-
-	notifiers[name] = n
-}
-
-// Run starts the Notifier service.
-func Run(config *config.NotifierConfig, datastore database.Datastore, stopper *utils.Stopper) {
 	defer stopper.End()
 
 	// Configure registered notifiers.
-	for notifierName, notifier := range notifiers {
+	for senderName, sender := range notification.Senders() {
-		if configured, err := notifier.Configure(config); configured {
+		if configured, err := sender.Configure(config); configured {
-			log.Infof("notifier '%s' configured\n", notifierName)
+			log.Infof("sender '%s' configured\n", senderName)
 		} else {
-			delete(notifiers, notifierName)
+			notification.UnregisterSender(senderName)
 			if err != nil {
-				log.Errorf("could not configure notifier '%s': %s", notifierName, err)
+				log.Errorf("could not configure notifier '%s': %s", senderName, err)
 			}
 		}
 	}
 
 	// Do not run the updater if there is no notifier enabled.
-	if len(notifiers) == 0 {
+	if len(notification.Senders()) == 0 {
 		log.Infof("notifier service is disabled")
 		return
 	}
@@ -124,8 +90,9 @@ func Run(config *config.NotifierConfig, datastore database.Datastore, stopper *u
 		go func() {
 			success, interrupted := handleTask(*notification, stopper, config.Attempts)
 			if success {
-				utils.PrometheusObserveTimeMilliseconds(promNotifierLatencyMilliseconds, notification.Created)
 				datastore.SetNotificationNotified(notification.Name)
+
+				promNotifierLatencyMilliseconds.Observe(float64(time.Since(notification.Created).Nanoseconds()) / float64(time.Millisecond))
 			}
 			if interrupted {
 				running = false
@@ -140,8 +107,8 @@ func Run(config *config.NotifierConfig, datastore database.Datastore, stopper *u
 			select {
 			case <-done:
 				break outer
-			case <-time.After(refreshLockDuration):
+			case <-time.After(notifierLockRefreshDuration):
-				datastore.Lock(notification.Name, whoAmI, lockDuration, true)
+				datastore.Lock(notification.Name, whoAmI, notifierLockDuration, true)
 			}
 		}
 	}
@@ -149,18 +116,18 @@ func Run(config *config.NotifierConfig, datastore database.Datastore, stopper *u
 	log.Info("notifier service stopped")
 }
 
-func findTask(datastore database.Datastore, renotifyInterval time.Duration, whoAmI string, stopper *utils.Stopper) *database.VulnerabilityNotification {
+func findTask(datastore database.Datastore, renotifyInterval time.Duration, whoAmI string, stopper *stopper.Stopper) *database.VulnerabilityNotification {
 	for {
 		// Find a notification to send.
 		notification, err := datastore.GetAvailableNotification(renotifyInterval)
 		if err != nil {
 			// There is no notification or an error occurred.
-			if err != cerrors.ErrNotFound {
+			if err != commonerr.ErrNotFound {
 				log.Warningf("could not get notification to send: %s", err)
 			}
 
 			// Wait.
-			if !stopper.Sleep(checkInterval) {
+			if !stopper.Sleep(notifierCheckInterval) {
 				return nil
 			}
 
@@ -168,39 +135,39 @@ func findTask(datastore database.Datastore, renotifyInterval time.Duration, whoA
 		}
 
 		// Lock the notification.
-		if hasLock, _ := datastore.Lock(notification.Name, whoAmI, lockDuration, false); hasLock {
+		if hasLock, _ := datastore.Lock(notification.Name, whoAmI, notifierLockDuration, false); hasLock {
 			log.Infof("found and locked a notification: %s", notification.Name)
 			return &notification
 		}
 	}
 }
 
-func handleTask(notification database.VulnerabilityNotification, st *utils.Stopper, maxAttempts int) (bool, bool) {
+func handleTask(n database.VulnerabilityNotification, st *stopper.Stopper, maxAttempts int) (bool, bool) {
 	// Send notification.
-	for notifierName, notifier := range notifiers {
+	for senderName, sender := range notification.Senders() {
 		var attempts int
 		var backOff time.Duration
 		for {
 			// Max attempts exceeded.
 			if attempts >= maxAttempts {
-				log.Infof("giving up on sending notification '%s' via notifier '%s': max attempts exceeded (%d)\n", notification.Name, notifierName, maxAttempts)
+				log.Infof("giving up on sending notification '%s' via sender '%s': max attempts exceeded (%d)\n", n.Name, senderName, maxAttempts)
 				return false, false
 			}
 
 			// Backoff.
 			if backOff > 0 {
-				log.Infof("waiting %v before retrying to send notification '%s' via notifier '%s' (Attempt %d / %d)\n", backOff, notification.Name, notifierName, attempts+1, maxAttempts)
+				log.Infof("waiting %v before retrying to send notification '%s' via sender '%s' (Attempt %d / %d)\n", backOff, n.Name, senderName, attempts+1, maxAttempts)
 				if !st.Sleep(backOff) {
 					return false, true
 				}
 			}
 
 			// Send using the current notifier.
-			if err := notifier.Send(notification); err != nil {
+			if err := sender.Send(n); err != nil {
 				// Send failed; increase attempts/backoff and retry.
-				promNotifierBackendErrorsTotal.WithLabelValues(notifierName).Inc()
+				promNotifierBackendErrorsTotal.WithLabelValues(senderName).Inc()
-				log.Errorf("could not send notification '%s' via notifier '%s': %v", notification.Name, notifierName, err)
+				log.Errorf("could not send notification '%s' via notifier '%s': %v", n.Name, senderName, err)
-				backOff = timeutil.ExpBackoff(backOff, maxBackOff)
+				backOff = timeutil.ExpBackoff(backOff, notifierMaxBackOff)
 				attempts++
 				continue
 			}
@@ -210,6 +177,6 @@ func handleTask(notification database.VulnerabilityNotification, st *utils.Stopp
 		}
 	}
 
-	log.Infof("successfully sent notification '%s'\n", notification.Name)
+	log.Infof("successfully sent notification '%s'\n", n.Name)
 	return true, false
 }

pkg/commonerr/errors.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,8 +12,9 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-// Package errors defines error types that are used in several modules
+// Package commonerr defines reusable error types common throughout the Clair
-package errors
+// codebase.
+package commonerr
 
 import "errors"
 

pkg/stopper/stopper.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package utils
+package stopper
 
 import (
 	"sync"

pkg/tarutil/tarutil.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,7 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package utils
+// Package tarutil implements some tar utility functions.
+package tarutil
 
 import (
 	"archive/tar"
@@ -29,28 +30,87 @@ import (
 
 var (
 	// ErrCouldNotExtract occurs when an extraction fails.
-	ErrCouldNotExtract = errors.New("utils: could not extract the archive")
+	ErrCouldNotExtract = errors.New("tarutil: could not extract the archive")
 
 	// ErrExtractedFileTooBig occurs when a file to extract is too big.
-	ErrExtractedFileTooBig = errors.New("utils: could not extract one or more files from the archive: file too big")
+	ErrExtractedFileTooBig = errors.New("tarutil: could not extract one or more files from the archive: file too big")
 
-	readLen = 6 // max bytes to sniff
+	// MaxExtractableFileSize enforces the maximum size of a single file within a
+	// tarball that will be extracted. This protects against malicious files that
+	// may used in an attempt to perform a Denial of Service attack.
+	MaxExtractableFileSize int64 = 200 * 1024 * 1024 // 200 MiB
 
+	readLen     = 6 // max bytes to sniff
 	gzipHeader  = []byte{0x1f, 0x8b}
 	bzip2Header = []byte{0x42, 0x5a, 0x68}
 	xzHeader    = []byte{0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00}
 )
 
-// XzReader is an io.ReadCloser which decompresses xz compressed data.
+// FilesMap is a map of files' paths to their contents.
+type FilesMap map[string][]byte
+
+// ExtractFiles decompresses and extracts only the specified files from an
+// io.Reader representing an archive.
+func ExtractFiles(r io.Reader, filenames []string) (FilesMap, error) {
+	data := make(map[string][]byte)
+
+	// Decompress the archive.
+	tr, err := NewTarReadCloser(r)
+	if err != nil {
+		return data, ErrCouldNotExtract
+	}
+	defer tr.Close()
+
+	// For each element in the archive
+	for {
+		hdr, err := tr.Next()
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			return data, ErrCouldNotExtract
+		}
+
+		// Get element filename
+		filename := hdr.Name
+		filename = strings.TrimPrefix(filename, "./")
+
+		// Determine if we should extract the element
+		toBeExtracted := false
+		for _, s := range filenames {
+			if strings.HasPrefix(filename, s) {
+				toBeExtracted = true
+				break
+			}
+		}
+
+		if toBeExtracted {
+			// File size limit
+			if hdr.Size > MaxExtractableFileSize {
+				return data, ErrExtractedFileTooBig
+			}
+
+			// Extract the element
+			if hdr.Typeflag == tar.TypeSymlink || hdr.Typeflag == tar.TypeLink || hdr.Typeflag == tar.TypeReg {
+				d, _ := ioutil.ReadAll(tr)
+				data[filename] = d
+			}
+		}
+	}
+
+	return data, nil
+}
+
+// XzReader implements io.ReadCloser for data compressed via `xz`.
 type XzReader struct {
 	io.ReadCloser
 	cmd     *exec.Cmd
 	closech chan error
 }
 
-// NewXzReader shells out to a command line xz executable (if
+// NewXzReader returns an io.ReadCloser by executing a command line `xz`
-// available) to decompress the given io.Reader using the xz
+// executable to decompress the provided io.Reader.
-// compression format and returns an *XzReader.
+//
 // It is the caller's responsibility to call Close on the XzReader when done.
 func NewXzReader(r io.Reader) (*XzReader, error) {
 	rpipe, wpipe := io.Pipe()
@@ -74,6 +134,7 @@ func NewXzReader(r io.Reader) (*XzReader, error) {
 	return &XzReader{rpipe, cmd, closech}, nil
 }
 
+// Close cleans up the resources used by an XzReader.
 func (r *XzReader) Close() error {
 	r.ReadCloser.Close()
 	r.cmd.Process.Kill()
@@ -88,72 +149,20 @@ type TarReadCloser struct {
 	io.Closer
 }
 
+// Close cleans up the resources used by a TarReadCloser.
 func (r *TarReadCloser) Close() error {
 	return r.Closer.Close()
 }
 
-// SelectivelyExtractArchive extracts the specified files and folders
+// NewTarReadCloser attempts to detect the compression algorithm for an
-// from targz data read from the given reader and store them in a map indexed by file paths
+// io.Reader and returns a TarReadCloser wrapping the Reader to transparently
-func SelectivelyExtractArchive(r io.Reader, prefix string, toExtract []string, maxFileSize int64) (map[string][]byte, error) {
+// decompress the contents.
-	data := make(map[string][]byte)
-
-	// Create a tar or tar/tar-gzip/tar-bzip2/tar-xz reader
-	tr, err := getTarReader(r)
-	if err != nil {
-		return data, ErrCouldNotExtract
-	}
-	defer tr.Close()
-
-	// For each element in the archive
-	for {
-		hdr, err := tr.Next()
-		if err == io.EOF {
-			break
-		}
-		if err != nil {
-			return data, ErrCouldNotExtract
-		}
-
-		// Get element filename
-		filename := hdr.Name
-		filename = strings.TrimPrefix(filename, "./")
-		if prefix != "" {
-			filename = strings.TrimPrefix(filename, prefix)
-		}
-
-		// Determine if we should extract the element
-		toBeExtracted := false
-		for _, s := range toExtract {
-			if strings.HasPrefix(filename, s) {
-				toBeExtracted = true
-				break
-			}
-		}
-
-		if toBeExtracted {
-			// File size limit
-			if maxFileSize > 0 && hdr.Size > maxFileSize {
-				return data, ErrExtractedFileTooBig
-			}
-
-			// Extract the element
-			if hdr.Typeflag == tar.TypeSymlink || hdr.Typeflag == tar.TypeLink || hdr.Typeflag == tar.TypeReg {
-				d, _ := ioutil.ReadAll(tr)
-				data[filename] = d
-			}
-		}
-	}
-
-	return data, nil
-}
-
-// getTarReader returns a TarReaderCloser associated with the specified io.Reader.
 //
 // Gzip/Bzip2/XZ detection is done by using the magic numbers:
 // Gzip: the first two bytes should be 0x1f and 0x8b. Defined in the RFC1952.
 // Bzip2: the first three bytes should be 0x42, 0x5a and 0x68. No RFC.
 // XZ: the first three bytes should be 0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00. No RFC.
-func getTarReader(r io.Reader) (*TarReadCloser, error) {
+func NewTarReadCloser(r io.Reader) (*TarReadCloser, error) {
 	br := bufio.NewReader(r)
 	header, err := br.Peek(readLen)
 	if err == nil {

pkg/tarutil/tarutil_test.go

@@ -0,0 +1,80 @@
+// Copyright 2017 clair authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package tarutil
+
+import (
+	"bytes"
+	"os"
+	"path/filepath"
+	"runtime"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+var testTarballs = []string{
+	"utils_test.tar",
+	"utils_test.tar.gz",
+	"utils_test.tar.bz2",
+	"utils_test.tar.xz",
+}
+
+func testfilepath(filename string) string {
+	_, path, _, _ := runtime.Caller(0)
+	testDataDir := "/testdata"
+	return filepath.Join(filepath.Dir(path), testDataDir, filename)
+}
+
+func TestExtract(t *testing.T) {
+	for _, filename := range testTarballs {
+		f, err := os.Open(testfilepath(filename))
+		assert.Nil(t, err)
+		defer f.Close()
+
+		data, err := ExtractFiles(f, []string{"test/"})
+		assert.Nil(t, err)
+
+		if c, n := data["test/test.txt"]; !n {
+			assert.Fail(t, "test/test.txt should have been extracted")
+		} else {
+			assert.NotEqual(t, 0, len(c) > 0, "test/test.txt file is empty")
+		}
+		if _, n := data["test.txt"]; n {
+			assert.Fail(t, "test.txt should not be extracted")
+		}
+	}
+}
+
+func TestExtractUncompressedData(t *testing.T) {
+	for _, filename := range testTarballs {
+		f, err := os.Open(testfilepath(filename))
+		assert.Nil(t, err)
+		defer f.Close()
+
+		_, err = ExtractFiles(bytes.NewReader([]byte("that string does not represent a tar or tar-gzip file")), []string{})
+		assert.Error(t, err, "Extracting uncompressed data should return an error")
+	}
+}
+
+func TestMaxExtractableFileSize(t *testing.T) {
+	for _, filename := range testTarballs {
+		f, err := os.Open(testfilepath(filename))
+		assert.Nil(t, err)
+		defer f.Close()
+		MaxExtractableFileSize = 50
+		_, err = ExtractFiles(f, []string{"test"})
+		assert.Equal(t, ErrExtractedFileTooBig, err)
+	}
+}

updater.go

@@ -1,4 +1,4 @@
-// Copyright 2015 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,9 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-// Package updater updates the vulnerability database periodically using
+package clair
-// the registered vulnerability fetchers.
-package updater
 
 import (
 	"math/rand"
@@ -22,25 +20,25 @@ import (
 	"sync"
 	"time"
 
-	"github.com/coreos/clair/config"
-	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/utils"
 	"github.com/coreos/pkg/capnslog"
 	"github.com/pborman/uuid"
 	"github.com/prometheus/client_golang/prometheus"
+
+	"github.com/coreos/clair/database"
+	"github.com/coreos/clair/ext/vulnmdsrc"
+	"github.com/coreos/clair/ext/vulnsrc"
+	"github.com/coreos/clair/pkg/stopper"
 )
 
 const (
-	flagName      = "updater/last"
+	updaterLastFlagName        = "updater/last"
-	notesFlagName = "updater/notes"
+	updaterLockName            = "updater"
-
+	updaterLockDuration        = updaterLockRefreshDuration + time.Minute*2
-	lockName            = "updater"
+	updaterLockRefreshDuration = time.Minute * 8
-	lockDuration        = refreshLockDuration + time.Minute*2
-	refreshLockDuration = time.Minute * 8
 )
 
 var (
-	log = capnslog.NewPackageLogger("github.com/coreos/clair", "updater")
+	log = capnslog.NewPackageLogger("github.com/coreos/clair", "clair")
 
 	promUpdaterErrorsTotal = prometheus.NewCounter(prometheus.CounterOpts{
 		Name: "clair_updater_errors_total",
@@ -64,8 +62,14 @@ func init() {
 	prometheus.MustRegister(promUpdaterNotesTotal)
 }
 
-// Run updates the vulnerability database at regular intervals.
+// UpdaterConfig is the configuration for the Updater service.
-func Run(config *config.UpdaterConfig, datastore database.Datastore, st *utils.Stopper) {
+type UpdaterConfig struct {
+	Interval time.Duration
+}
+
+// RunUpdater begins a process that updates the vulnerability database at
+// regular intervals.
+func RunUpdater(config *UpdaterConfig, datastore database.Datastore, st *stopper.Stopper) {
 	defer st.End()
 
 	// Do not run the updater if there is no config or if the interval is 0.
@@ -95,12 +99,12 @@ func Run(config *config.UpdaterConfig, datastore database.Datastore, st *utils.S
 		if nextUpdate.Before(time.Now().UTC()) {
 			// Attempt to get a lock on the the update.
 			log.Debug("attempting to obtain update lock")
-			hasLock, hasLockUntil := datastore.Lock(lockName, whoAmI, lockDuration, false)
+			hasLock, hasLockUntil := datastore.Lock(updaterLockName, whoAmI, updaterLockDuration, false)
 			if hasLock {
 				// Launch update in a new go routine.
 				doneC := make(chan bool, 1)
 				go func() {
-					Update(datastore, firstUpdate)
+					update(datastore, firstUpdate)
 					doneC <- true
 				}()
 
@@ -108,23 +112,23 @@ func Run(config *config.UpdaterConfig, datastore database.Datastore, st *utils.S
 					select {
 					case <-doneC:
 						done = true
-					case <-time.After(refreshLockDuration):
+					case <-time.After(updaterLockRefreshDuration):
 						// Refresh the lock until the update is done.
-						datastore.Lock(lockName, whoAmI, lockDuration, true)
+						datastore.Lock(updaterLockName, whoAmI, updaterLockDuration, true)
 					case <-st.Chan():
 						stop = true
 					}
 				}
 
 				// Unlock the update.
-				datastore.Unlock(lockName, whoAmI)
+				datastore.Unlock(updaterLockName, whoAmI)
 
 				if stop {
 					break
 				}
 				continue
 			} else {
-				lockOwner, lockExpiration, err := datastore.FindLock(lockName)
+				lockOwner, lockExpiration, err := datastore.FindLock(updaterLockName)
 				if err != nil {
 					log.Debug("update lock is already taken")
 					nextUpdate = hasLockUntil
@@ -147,19 +151,19 @@ func Run(config *config.UpdaterConfig, datastore database.Datastore, st *utils.S
 	}
 
 	// Clean resources.
-	for _, metadataFetcher := range metadataFetchers {
+	for _, appenders := range vulnmdsrc.Appenders() {
-		metadataFetcher.Clean()
+		appenders.Clean()
 	}
-	for _, fetcher := range fetchers {
+	for _, updaters := range vulnsrc.Updaters() {
-		fetcher.Clean()
+		updaters.Clean()
 	}
 
 	log.Info("updater service stopped")
 }
 
-// Update fetches all the vulnerabilities from the registered fetchers, upserts
+// update fetches all the vulnerabilities from the registered fetchers, upserts
 // them into the database and then sends notifications.
-func Update(datastore database.Datastore, firstUpdate bool) {
+func update(datastore database.Datastore, firstUpdate bool) {
 	defer setUpdaterDuration(time.Now())
 
 	log.Info("updating vulnerabilities")
@@ -190,7 +194,7 @@ func Update(datastore database.Datastore, firstUpdate bool) {
 
 	// Update last successful update if every fetchers worked properly.
 	if status {
-		datastore.InsertKeyValue(flagName, strconv.FormatInt(time.Now().UTC().Unix(), 10))
+		datastore.InsertKeyValue(updaterLastFlagName, strconv.FormatInt(time.Now().UTC().Unix(), 10))
 	}
 
 	log.Info("update finished")
@@ -209,10 +213,10 @@ func fetch(datastore database.Datastore) (bool, []database.Vulnerability, map[st
 
 	// Fetch updates in parallel.
 	log.Info("fetching vulnerability updates")
-	var responseC = make(chan *FetcherResponse, 0)
+	var responseC = make(chan *vulnsrc.UpdateResponse, 0)
-	for n, f := range fetchers {
+	for n, u := range vulnsrc.Updaters() {
-		go func(name string, fetcher Fetcher) {
+		go func(name string, u vulnsrc.Updater) {
-			response, err := fetcher.FetchUpdate(datastore)
+			response, err := u.Update(datastore)
 			if err != nil {
 				promUpdaterErrorsTotal.Inc()
 				log.Errorf("an error occured when fetching update '%s': %s.", name, err)
@@ -222,11 +226,11 @@ func fetch(datastore database.Datastore) (bool, []database.Vulnerability, map[st
 			}
 
 			responseC <- &response
-		}(n, f)
+		}(n, u)
 	}
 
 	// Collect results of updates.
-	for i := 0; i < len(fetchers); i++ {
+	for i := 0; i < len(vulnsrc.Updaters()); i++ {
 		resp := <-responseC
 		if resp != nil {
 			vulnerabilities = append(vulnerabilities, doVulnerabilitiesNamespacing(resp.Vulnerabilities)...)
@@ -243,42 +247,43 @@ func fetch(datastore database.Datastore) (bool, []database.Vulnerability, map[st
 
 // Add metadata to the specified vulnerabilities using the registered MetadataFetchers, in parallel.
 func addMetadata(datastore database.Datastore, vulnerabilities []database.Vulnerability) []database.Vulnerability {
-	if len(metadataFetchers) == 0 {
+	if len(vulnmdsrc.Appenders()) == 0 {
 		return vulnerabilities
 	}
 
 	log.Info("adding metadata to vulnerabilities")
 
-	// Wrap vulnerabilities in VulnerabilityWithLock.
+	// Add a mutex to each vulnerability to ensure that only one appender at a
-	// It ensures that only one metadata fetcher at a time can modify the Metadata map.
+	// time can modify the vulnerability's Metadata map.
-	vulnerabilitiesWithLocks := make([]*VulnerabilityWithLock, 0, len(vulnerabilities))
+	lockableVulnerabilities := make([]*lockableVulnerability, 0, len(vulnerabilities))
 	for i := 0; i < len(vulnerabilities); i++ {
-		vulnerabilitiesWithLocks = append(vulnerabilitiesWithLocks, &VulnerabilityWithLock{
+		lockableVulnerabilities = append(lockableVulnerabilities, &lockableVulnerability{
 			Vulnerability: &vulnerabilities[i],
 		})
 	}
 
 	var wg sync.WaitGroup
-	wg.Add(len(metadataFetchers))
+	wg.Add(len(vulnmdsrc.Appenders()))
 
-	for n, f := range metadataFetchers {
+	for n, a := range vulnmdsrc.Appenders() {
-		go func(name string, metadataFetcher MetadataFetcher) {
+		go func(name string, appender vulnmdsrc.Appender) {
 			defer wg.Done()
 
-			// Load the metadata fetcher.
+			// Build up a metadata cache.
-			if err := metadataFetcher.Load(datastore); err != nil {
+			if err := appender.BuildCache(datastore); err != nil {
 				promUpdaterErrorsTotal.Inc()
 				log.Errorf("an error occured when loading metadata fetcher '%s': %s.", name, err)
 				return
 			}
 
-			// Add metadata to each vulnerability.
+			// Append vulnerability metadata  to each vulnerability.
-			for _, vulnerability := range vulnerabilitiesWithLocks {
+			for _, vulnerability := range lockableVulnerabilities {
-				metadataFetcher.AddMetadata(vulnerability)
+				appender.Append(vulnerability.Name, vulnerability.appendFunc)
 			}
 
-			metadataFetcher.Unload()
+			// Purge the metadata cache.
-		}(n, f)
+			appender.PurgeCache()
+		}(n, a)
 	}
 
 	wg.Wait()
@@ -287,7 +292,7 @@ func addMetadata(datastore database.Datastore, vulnerabilities []database.Vulner
 }
 
 func getLastUpdate(datastore database.Datastore) (time.Time, bool, error) {
-	lastUpdateTSS, err := datastore.GetKeyValue(flagName)
+	lastUpdateTSS, err := datastore.GetKeyValue(updaterLastFlagName)
 	if err != nil {
 		return time.Time{}, false, err
 	}
@@ -305,6 +310,29 @@ func getLastUpdate(datastore database.Datastore) (time.Time, bool, error) {
 	return time.Unix(lastUpdateTS, 0).UTC(), false, nil
 }
 
+type lockableVulnerability struct {
+	*database.Vulnerability
+	sync.Mutex
+}
+
+func (lv *lockableVulnerability) appendFunc(metadataKey string, metadata interface{}, severity database.Severity) {
+	lv.Lock()
+	defer lv.Unlock()
+
+	// If necessary, initialize the metadata map for the vulnerability.
+	if lv.Metadata == nil {
+		lv.Metadata = make(map[string]interface{})
+	}
+
+	// Append the metadata.
+	lv.Metadata[metadataKey] = metadata
+
+	// If necessary, provide a severity for the vulnerability.
+	if lv.Severity == database.UnknownSeverity {
+		lv.Severity = severity
+	}
+}
+
 // doVulnerabilitiesNamespacing takes Vulnerabilities that don't have a Namespace and split them
 // into multiple vulnerabilities that have a Namespace and only contains the FixedIn
 // FeatureVersions corresponding to their Namespace.

updater/fetchers.go

@@ -1,56 +0,0 @@
-// Copyright 2015 clair authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package updater
-
-import "github.com/coreos/clair/database"
-
-var fetchers = make(map[string]Fetcher)
-
-// Fetcher represents anything that can fetch vulnerabilities.
-type Fetcher interface {
-	// FetchUpdate gets vulnerability updates.
-	FetchUpdate(database.Datastore) (FetcherResponse, error)
-
-	// Clean deletes any allocated resources.
-	// It is invoked when Clair stops.
-	Clean()
-}
-
-// FetcherResponse represents the sum of results of an update.
-type FetcherResponse struct {
-	FlagName        string
-	FlagValue       string
-	Notes           []string
-	Vulnerabilities []database.Vulnerability
-}
-
-// RegisterFetcher makes a Fetcher available by the provided name.
-// If Register is called twice with the same name or if driver is nil,
-// it panics.
-func RegisterFetcher(name string, f Fetcher) {
-	if name == "" {
-		panic("updater: could not register a Fetcher with an empty name")
-	}
-
-	if f == nil {
-		panic("updater: could not register a nil Fetcher")
-	}
-
-	if _, dup := fetchers[name]; dup {
-		panic("updater: RegisterFetcher called twice for " + name)
-	}
-
-	fetchers[name] = f
-}

updater/metadata_fetchers.go

@@ -1,64 +0,0 @@
-// Copyright 2015 clair authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package updater
-
-import (
-	"sync"
-
-	"github.com/coreos/clair/database"
-)
-
-var metadataFetchers = make(map[string]MetadataFetcher)
-
-type VulnerabilityWithLock struct {
-	*database.Vulnerability
-	Lock sync.Mutex
-}
-
-// MetadataFetcher
-type MetadataFetcher interface {
-	// Load runs right before the Updater calls AddMetadata for each vulnerabilities.
-	Load(database.Datastore) error
-
-	// AddMetadata adds metadata to the given database.Vulnerability.
-	// It is expected that the fetcher uses .Lock.Lock() when manipulating the Metadata map.
-	AddMetadata(*VulnerabilityWithLock) error
-
-	// Unload runs right after the Updater finished calling AddMetadata for every vulnerabilities.
-	Unload()
-
-	// Clean deletes any allocated resources.
-	// It is invoked when Clair stops.
-	Clean()
-}
-
-// RegisterFetcher makes a Fetcher available by the provided name.
-// If Register is called twice with the same name or if driver is nil,
-// it panics.
-func RegisterMetadataFetcher(name string, f MetadataFetcher) {
-	if name == "" {
-		panic("updater: could not register a MetadataFetcher with an empty name")
-	}
-
-	if f == nil {
-		panic("updater: could not register a nil MetadataFetcher")
-	}
-
-	if _, dup := fetchers[name]; dup {
-		panic("updater: RegisterMetadataFetcher called twice for " + name)
-	}
-
-	metadataFetchers[name] = f
-}

updater/metadata_fetchers/nvd/nested_read_closer.go

@@ -1,19 +0,0 @@
-package nvd
-
-import "io"
-
-// NestedReadCloser wraps an io.Reader and implements io.ReadCloser by closing every embed
-// io.ReadCloser.
-// It allows chaining io.ReadCloser together and still keep the ability to close them all in a
-// simple manner.
-type NestedReadCloser struct {
-	io.Reader
-	NestedReadClosers []io.ReadCloser
-}
-
-// Close closes the gzip.Reader and the underlying io.ReadCloser.
-func (nrc *NestedReadCloser) Close() {
-	for _, nestedReadCloser := range nrc.NestedReadClosers {
-		nestedReadCloser.Close()
-	}
-}

updater_test.go

@@ -1,4 +1,4 @@
-// Copyright 2016 clair authors
+// Copyright 2017 clair authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-package updater
+package clair
 
 import (
 	"fmt"

utils/http/http.go

@@ -1,77 +0,0 @@
-// Copyright 2015 clair authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-// Package http provides utility functions for HTTP servers and clients.
-package http
-
-import (
-	"encoding/json"
-	"io"
-	"net/http"
-
-	"github.com/coreos/clair/database"
-	"github.com/coreos/clair/utils"
-	cerrors "github.com/coreos/clair/utils/errors"
-	"github.com/coreos/clair/worker"
-)
-
-// MaxBodySize is the maximum number of bytes that ParseHTTPBody reads from an http.Request.Body.
-const MaxBodySize int64 = 1048576
-
-// WriteHTTP writes a JSON-encoded object to a http.ResponseWriter, as well as
-// a HTTP status code.
-func WriteHTTP(w http.ResponseWriter, httpStatus int, v interface{}) {
-	w.WriteHeader(httpStatus)
-	if v != nil {
-		w.Header().Set("Content-Type", "application/json; charset=utf-8")
-		result, _ := json.Marshal(v)
-		w.Write(result)
-	}
-}
-
-// WriteHTTPError writes an error, wrapped in the Message field of a JSON-encoded
-// object to a http.ResponseWriter, as well as a HTTP status code.
-// If the status code is 0, handleError tries to guess the proper HTTP status
-// code from the error type.
-func WriteHTTPError(w http.ResponseWriter, httpStatus int, err error) {
-	if httpStatus == 0 {
-		httpStatus = http.StatusInternalServerError
-		// Try to guess the http status code from the error type
-		if _, isBadRequestError := err.(*cerrors.ErrBadRequest); isBadRequestError {
-			httpStatus = http.StatusBadRequest
-		} else {
-			switch err {
-			case cerrors.ErrNotFound:
-				httpStatus = http.StatusNotFound
-			case database.ErrBackendException:
-				httpStatus = http.StatusServiceUnavailable
-			case worker.ErrParentUnknown, worker.ErrUnsupported, utils.ErrCouldNotExtract, utils.ErrExtractedFileTooBig:
-				httpStatus = http.StatusBadRequest
-			}
-		}
-	}
-
-	WriteHTTP(w, httpStatus, struct{ Message string }{Message: err.Error()})
-}
-
-// ParseHTTPBody reads a JSON-encoded body from a http.Request and unmarshals it
-// into the provided object.
-func ParseHTTPBody(r *http.Request, v interface{}) (int, error) {
-	defer r.Body.Close()
-	err := json.NewDecoder(io.LimitReader(r.Body, MaxBodySize)).Decode(v)
-	if err != nil {
-		return http.StatusUnsupportedMediaType, err
-	}
-	return 0, nil
-}

utils/prometheus.go

@@ -1,13 +0,0 @@
-package utils
-
-import (
-	"time"
-
-	"github.com/prometheus/client_golang/prometheus"
-)
-
-// PrometheusObserveTimeMilliseconds observes the elapsed time since start, in milliseconds,
-// on the specified Prometheus Histogram.
-func PrometheusObserveTimeMilliseconds(h prometheus.Histogram, start time.Time) {
-	h.Observe(float64(time.Since(start).Nanoseconds()) / float64(time.Millisecond))
-}

utils/string.go

@@ -1,75 +0,0 @@
-// Copyright 2015 clair authors
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package utils
-
-import "regexp"
-
-var urlParametersRegexp = regexp.MustCompile(`(\?|\&)([^=]+)\=([^ &]+)`)
-
-// CleanURL removes all parameters from an URL.
-func CleanURL(str string) string {
-	return urlParametersRegexp.ReplaceAllString(str, "")
-}
-
-// Contains looks for a string into an array of strings and returns whether
-// the string exists.
-func Contains(needle string, haystack []string) bool {
-	for _, h := range haystack {
-		if h == needle {
-			return true
-		}
-	}
-	return false
-}
-
-// CompareStringLists returns the strings that are present in X but not in Y.
-func CompareStringLists(X, Y []string) []string {
-	m := make(map[string]bool)
-
-	for _, y := range Y {
-		m[y] = true
-	}
-
-	diff := []string{}
-	for _, x := range X {
-		if m[x] {
-			continue
-		}
-
-		diff = append(diff, x)
-		m[x] = true
-	}
-
-	return diff
-}
-
-// CompareStringListsInBoth returns the strings that are present in both X and Y.
-func CompareStringListsInBoth(X, Y []string) []string {
-	m := make(map[string]struct{})
-
-	for _, y := range Y {
-		m[y] = struct{}{}
-	}
-
-	diff := []string{}
-	for _, x := range X {
-		if _, e := m[x]; e {
-			diff = append(diff, x)
-			delete(m, x)
-		}
-	}
-
-	return diff
-}