arno/clair
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Merge pull request #305 from jzelinskie/ext

Browse Source 
Refactor all extendable code into ext/
This commit is contained in:
Jimmy Zelinskie 2017-02-06 15:39:15 -08:00 committed by GitHub
commit eb5be92305
113 changed files with 2274 additions and 2830 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Documentation/api_v1.md
View File

@ -19,7 +19,7 @@
- [DELETE](#delete-namespacesnsnamevulnerabilitiesvulnnamefixesfeaturename)
- [Notifications](#notifications)
- [GET](#get-notificationsname)
- [DELETE](#delete-notificationname)
- [DELETE](#delete-notificationsname)
## Error Handling

35
README.md
View File

@ -14,15 +14,15 @@ Please use [releases] instead of the `master` branch in order to get stable bina
Clair is an open source project for the static analysis of vulnerabilities in [appc] and [docker] containers.
Vulnerability data is continuously imported from a known set of sources and correlated with the indexed contents of container images in order to produce lists of vulnerabilities that threaten a container.
When vulnerability data changes upstream, the previous state and new state of the vulnerability along with the images they affect can be sent via webhook to a configured endpoint.
All major components can be [customized programmatically] at compile-time without forking the project.
When vulnerability data changes upstream, a notification can be delivered, and the API queried to provide the previous state and new state of the vulnerability along with the images affected by both.
All major components can be [extended programmatically] at compile-time without forking the project.
Our goal is to enable a more transparent view of the security of container-based infrastructure.
Thus, the project was named `Clair` after the French term which translates to *clear*, *bright*, *transparent*.
[appc]: https://github.com/appc/spec
[docker]: https://github.com/docker/docker/blob/master/image/spec/v1.md
[customized programmatically]: #customization
[extended programmatically]: #customization
[releases]: https://github.com/coreos/clair/releases
## Common Use Cases
@ -127,7 +127,8 @@ While container images for every releases are available at [quay.io/repository/c
## Documentation
The latest stable documentation can be found [on the CoreOS website]. Documentation for the current branch can be found [inside the Documentation directory][docs-dir] at the root of the project's source code.
The latest stable documentation can be found [on the CoreOS website].
Documentation for the current branch can be found [inside the Documentation directory][docs-dir] at the root of the project's source code.
[on the CoreOS website]: https://coreos.com/clair/docs/latest/
[docs-dir]: /Documentation
@ -140,10 +141,10 @@ The latest stable documentation can be found [on the CoreOS website]. Documentat
- *Image* - a tarball of the contents of a container
- *Layer* - an *appc* or *Docker* image that may or maybe not be dependent on another image
- *Detector* - a Go package that identifies the content, *namespaces* and *features* from a *layer*
- *Namespace* - a context around *features* and *vulnerabilities* (e.g. an operating system)
- *Feature* - anything that when present could be an indication of a *vulnerability* (e.g. the presence of a file or an installed software package)
- *Fetcher* - a Go package that tracks an upstream vulnerability database and imports them into Clair
- *Feature Namespace* - a context around *features* and *vulnerabilities* (e.g. an operating system)
- *Vulnerability Updater* - a Go package that tracks upstream vulnerability data and imports them into Clair
- *Vulnerability Metadata Appender* - a Go package that tracks upstream vulnerability metadata and appends them into vulnerabilities managed by Clair
### Vulnerability Analysis
@ -164,13 +165,13 @@ By indexing the features of an image into the database, images only need to be r
| [Red Hat Security Data] | CentOS 5, 6, 7 namespaces | [rpm] | [CVRF] |
| [Oracle Linux Security Data] | Oracle Linux 5, 6, 7 namespaces | [rpm] | [CVRF] |
| [Alpine SecDB] | Alpine 3.3, Alpine 3.4 namespaces | [apk] | [MIT] |
| [NVD] | Generic Vulnerability Metadata | N/A | [Public Domain] |
| [NIST NVD] | Generic Vulnerability Metadata | N/A | [Public Domain] |
[Debian Security Bug Tracker]: https://security-tracker.debian.org/tracker
[Ubuntu CVE Tracker]: https://launchpad.net/ubuntu-cve-tracker
[Red Hat Security Data]: https://www.redhat.com/security/data/metrics
[Oracle Linux Security Data]: https://linux.oracle.com/security/
[NVD]: https://nvd.nist.gov
[NIST NVD]: https://nvd.nist.gov
[dpkg]: https://en.wikipedia.org/wiki/dpkg
[rpm]: http://www.rpm.org
[Debian]: https://www.debian.org/license
@ -185,21 +186,13 @@ By indexing the features of an image into the database, images only need to be r
### Customization
The major components of Clair are all programmatically extensible in the same way Go's standard [database/sql] package is extensible.
Everything extendable is located in the `ext` directory.
Custom behavior can be accomplished by creating a package that contains a type that implements an interface declared in Clair and registering that interface in [init()]. To expose the new behavior, unqualified imports to the package must be added in your [main.go], which should then start Clair using `Boot(*config.Config)`.
Custom behavior can be accomplished by creating a package that contains a type that implements an interface declared in Clair and registering that interface in [init()].
To expose the new behavior, unqualified imports to the package must be added in your own custom [main.go], which should then start Clair using `Boot(*config.Config)`.
The following interfaces can have custom implementations registered via [init()] at compile time:
- `Datastore` - the backing storage
- `Notifier` - the means by which endpoints are notified of vulnerability changes
- `Fetcher` - the sources of vulnerability data that is automatically imported
- `MetadataFetcher` - the sources of vulnerability metadata that is automatically added to known vulnerabilities
- `DataDetector` - the means by which contents of an image are detected
- `FeatureDetector` - the means by which features are identified from a layer
- `NamespaceDetector` - the means by which a namespace is identified from a layer
[init()]: https://golang.org/doc/effective_go.html#init
[database/sql]: https://godoc.org/database/sql
[init()]: https://golang.org/doc/effective_go.html#init
[main.go]: https://github.com/coreos/clair/blob/master/cmd/clair/main.go
## Related Links

46
api/api.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -23,29 +23,37 @@ import (
"strconv"
"time"
"github.com/coreos/pkg/capnslog"
"github.com/tylerb/graceful"
"github.com/coreos/clair/api/context"
"github.com/coreos/clair/config"
"github.com/coreos/clair/utils"
"github.com/coreos/pkg/capnslog"
"github.com/coreos/clair/database"
"github.com/coreos/clair/pkg/stopper"
)
const timeoutResponse = `{"Error":{"Message":"Clair failed to respond within the configured timeout window.","Type":"Timeout"}}`
var log = capnslog.NewPackageLogger("github.com/coreos/clair", "api")
func Run(config *config.APIConfig, ctx *context.RouteContext, st *utils.Stopper) {
// Config is the configuration for the API service.
type Config struct {
Port int
HealthPort int
Timeout time.Duration
PaginationKey string
CertFile, KeyFile, CAFile string
}
func Run(cfg *Config, store database.Datastore, st *stopper.Stopper) {
defer st.End()
// Do not run the API service if there is no config.
if config == nil {
if cfg == nil {
log.Infof("main API service is disabled.")
return
}
log.Infof("starting main API on port %d.", config.Port)
log.Infof("starting main API on port %d.", cfg.Port)
tlsConfig, err := tlsClientConfig(config.CAFile)
tlsConfig, err := tlsClientConfig(cfg.CAFile)
if err != nil {
log.Fatalf("could not initialize client cert authentication: %s\n", err)
}
@ -57,33 +65,33 @@ func Run(config *config.APIConfig, ctx *context.RouteContext, st *utils.Stopper)
Timeout: 0, // Already handled by our TimeOut middleware
NoSignalHandling: true, // We want to use our own Stopper
Server: &http.Server{
Addr: ":" + strconv.Itoa(config.Port),
Addr: ":" + strconv.Itoa(cfg.Port),
TLSConfig: tlsConfig,
Handler: http.TimeoutHandler(newAPIHandler(ctx), config.Timeout, timeoutResponse),
Handler: http.TimeoutHandler(newAPIHandler(cfg, store), cfg.Timeout, timeoutResponse),
},
}
listenAndServeWithStopper(srv, st, config.CertFile, config.KeyFile)
listenAndServeWithStopper(srv, st, cfg.CertFile, cfg.KeyFile)
log.Info("main API stopped")
}
func RunHealth(config *config.APIConfig, ctx *context.RouteContext, st *utils.Stopper) {
func RunHealth(cfg *Config, store database.Datastore, st *stopper.Stopper) {
defer st.End()
// Do not run the API service if there is no config.
if config == nil {
if cfg == nil {
log.Infof("health API service is disabled.")
return
}
log.Infof("starting health API on port %d.", config.HealthPort)
log.Infof("starting health API on port %d.", cfg.HealthPort)
srv := &graceful.Server{
Timeout: 10 * time.Second, // Interrupt health checks when stopping
NoSignalHandling: true, // We want to use our own Stopper
Server: &http.Server{
Addr: ":" + strconv.Itoa(config.HealthPort),
Handler: http.TimeoutHandler(newHealthHandler(ctx), config.Timeout, timeoutResponse),
Addr: ":" + strconv.Itoa(cfg.HealthPort),
Handler: http.TimeoutHandler(newHealthHandler(store), cfg.Timeout, timeoutResponse),
},
}
@ -94,8 +102,8 @@ func RunHealth(config *config.APIConfig, ctx *context.RouteContext, st *utils.St
// listenAndServeWithStopper wraps graceful.Server's
// ListenAndServe/ListenAndServeTLS and adds the ability to interrupt them with
// the provided utils.Stopper
func listenAndServeWithStopper(srv *graceful.Server, st *utils.Stopper, certFile, keyFile string) {
// the provided stopper.Stopper.
func listenAndServeWithStopper(srv *graceful.Server, st *stopper.Stopper, certFile, keyFile string) {
go func() {
<-st.Chan()
srv.Stop(0)

64
api/context/context.go
View File

@ -1,64 +0,0 @@
// Copyright 2015 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package context
import (
"net/http"
"strconv"
"time"
"github.com/coreos/pkg/capnslog"
"github.com/julienschmidt/httprouter"
"github.com/prometheus/client_golang/prometheus"
"github.com/coreos/clair/config"
"github.com/coreos/clair/database"
"github.com/coreos/clair/utils"
)
var (
log = capnslog.NewPackageLogger("github.com/coreos/clair", "api")
promResponseDurationMilliseconds = prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "clair_api_response_duration_milliseconds",
Help: "The duration of time it takes to receieve and write a response to an API request",
Buckets: prometheus.ExponentialBuckets(9.375, 2, 10),
}, []string{"route", "code"})
)
func init() {
prometheus.MustRegister(promResponseDurationMilliseconds)
}
type Handler func(http.ResponseWriter, *http.Request, httprouter.Params, *RouteContext) (route string, status int)
func HTTPHandler(handler Handler, ctx *RouteContext) httprouter.Handle {
return func(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
start := time.Now()
route, status := handler(w, r, p, ctx)
statusStr := strconv.Itoa(status)
if status == 0 {
statusStr = "???"
}
utils.PrometheusObserveTimeMilliseconds(promResponseDurationMilliseconds.WithLabelValues(route, statusStr), start)
log.Infof("%s \"%s %s\" %s (%s)", r.RemoteAddr, r.Method, r.RequestURI, statusStr, time.Since(start))
}
}
type RouteContext struct {
Store database.Datastore
Config *config.APIConfig
}

19
api/router.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -20,8 +20,8 @@ import (
"github.com/julienschmidt/httprouter"
"github.com/coreos/clair/api/context"
"github.com/coreos/clair/api/v1"
"github.com/coreos/clair/database"
)
// router is an HTTP router that forwards requests to the appropriate sub-router
@ -31,9 +31,9 @@ type router map[string]*httprouter.Router
// Let's hope we never have more than 99 API versions.
const apiVersionLength = len("v99")
func newAPIHandler(ctx *context.RouteContext) http.Handler {
func newAPIHandler(cfg *Config, store database.Datastore) http.Handler {
router := make(router)
router["/v1"] = v1.NewRouter(ctx)
router["/v1"] = v1.NewRouter(store, cfg.PaginationKey)
return router
}
@ -56,21 +56,22 @@ func (rtr router) ServeHTTP(w http.ResponseWriter, r *http.Request) {
http.NotFound(w, r)
}
func newHealthHandler(ctx *context.RouteContext) http.Handler {
func newHealthHandler(store database.Datastore) http.Handler {
router := httprouter.New()
router.GET("/health", context.HTTPHandler(getHealth, ctx))
router.GET("/health", healthHandler(store))
return router
}
func getHealth(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func healthHandler(store database.Datastore) httprouter.Handle {
return func(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
header := w.Header()
header.Set("Server", "clair")
status := http.StatusInternalServerError
if ctx.Store.Ping() {
if store.Ping() {
status = http.StatusOK
}
w.WriteHeader(status)
return "health", status
}
}

9
api/v1/models.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -26,7 +26,6 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/utils/types"
)
var log = capnslog.NewPackageLogger("github.com/coreos/clair", "v1")
@ -109,9 +108,9 @@ type Vulnerability struct {
}
func (v Vulnerability) DatabaseModel() (database.Vulnerability, error) {
severity := types.Priority(v.Severity)
if !severity.IsValid() {
return database.Vulnerability{}, errors.New("Invalid severity")
severity, err := database.NewSeverity(v.Severity)
if err != nil {
return database.Vulnerability{}, err
}
var dbFeatures []database.FeatureVersion

78
api/v1/router.go
View File

@ -16,41 +16,83 @@
package v1
import (
"github.com/julienschmidt/httprouter"
"net/http"
"strconv"
"time"
"github.com/coreos/clair/api/context"
"github.com/julienschmidt/httprouter"
"github.com/prometheus/client_golang/prometheus"
"github.com/coreos/clair/database"
)
var (
promResponseDurationMilliseconds = prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "clair_api_response_duration_milliseconds",
Help: "The duration of time it takes to receieve and write a response to an API request",
Buckets: prometheus.ExponentialBuckets(9.375, 2, 10),
}, []string{"route", "code"})
)
func init() {
prometheus.MustRegister(promResponseDurationMilliseconds)
}
type handler func(http.ResponseWriter, *http.Request, httprouter.Params, *context) (route string, status int)
func httpHandler(h handler, ctx *context) httprouter.Handle {
return func(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
start := time.Now()
route, status := h(w, r, p, ctx)
statusStr := strconv.Itoa(status)
if status == 0 {
statusStr = "???"
}
promResponseDurationMilliseconds.
WithLabelValues(route, statusStr).
Observe(float64(time.Since(start).Nanoseconds()) / float64(time.Millisecond))
log.Infof("%s \"%s %s\" %s (%s)", r.RemoteAddr, r.Method, r.RequestURI, statusStr, time.Since(start))
}
}
type context struct {
Store database.Datastore
PaginationKey string
}
// NewRouter creates an HTTP router for version 1 of the Clair API.
func NewRouter(ctx *context.RouteContext) *httprouter.Router {
func NewRouter(store database.Datastore, paginationKey string) *httprouter.Router {
router := httprouter.New()
ctx := &context{store, paginationKey}
// Layers
router.POST("/layers", context.HTTPHandler(postLayer, ctx))
router.GET("/layers/:layerName", context.HTTPHandler(getLayer, ctx))
router.DELETE("/layers/:layerName", context.HTTPHandler(deleteLayer, ctx))
router.POST("/layers", httpHandler(postLayer, ctx))
router.GET("/layers/:layerName", httpHandler(getLayer, ctx))
router.DELETE("/layers/:layerName", httpHandler(deleteLayer, ctx))
// Namespaces
router.GET("/namespaces", context.HTTPHandler(getNamespaces, ctx))
router.GET("/namespaces", httpHandler(getNamespaces, ctx))
// Vulnerabilities
router.GET("/namespaces/:namespaceName/vulnerabilities", context.HTTPHandler(getVulnerabilities, ctx))
router.POST("/namespaces/:namespaceName/vulnerabilities", context.HTTPHandler(postVulnerability, ctx))
router.GET("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName", context.HTTPHandler(getVulnerability, ctx))
router.PUT("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName", context.HTTPHandler(putVulnerability, ctx))
router.DELETE("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName", context.HTTPHandler(deleteVulnerability, ctx))
router.GET("/namespaces/:namespaceName/vulnerabilities", httpHandler(getVulnerabilities, ctx))
router.POST("/namespaces/:namespaceName/vulnerabilities", httpHandler(postVulnerability, ctx))
router.GET("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName", httpHandler(getVulnerability, ctx))
router.PUT("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName", httpHandler(putVulnerability, ctx))
router.DELETE("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName", httpHandler(deleteVulnerability, ctx))
// Fixes
router.GET("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName/fixes", context.HTTPHandler(getFixes, ctx))
router.PUT("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName/fixes/:fixName", context.HTTPHandler(putFix, ctx))
router.DELETE("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName/fixes/:fixName", context.HTTPHandler(deleteFix, ctx))
router.GET("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName/fixes", httpHandler(getFixes, ctx))
router.PUT("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName/fixes/:fixName", httpHandler(putFix, ctx))
router.DELETE("/namespaces/:namespaceName/vulnerabilities/:vulnerabilityName/fixes/:fixName", httpHandler(deleteFix, ctx))
// Notifications
router.GET("/notifications/:notificationName", context.HTTPHandler(getNotification, ctx))
router.DELETE("/notifications/:notificationName", context.HTTPHandler(deleteNotification, ctx))
router.GET("/notifications/:notificationName", httpHandler(getNotification, ctx))
router.DELETE("/notifications/:notificationName", httpHandler(deleteNotification, ctx))
// Metrics
router.GET("/metrics", context.HTTPHandler(getMetrics, ctx))
router.GET("/metrics", httpHandler(getMetrics, ctx))
return router
}

85
api/v1/routes.go
View File

@ -25,11 +25,10 @@ import (
"github.com/julienschmidt/httprouter"
"github.com/prometheus/client_golang/prometheus"
"github.com/coreos/clair/api/context"
"github.com/coreos/clair"
"github.com/coreos/clair/database"
"github.com/coreos/clair/utils"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/worker"
"github.com/coreos/clair/pkg/commonerr"
"github.com/coreos/clair/pkg/tarutil"
)
const (
@ -96,7 +95,7 @@ func writeResponse(w http.ResponseWriter, r *http.Request, status int, resp inte
}
}
func postLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func postLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
request := LayerEnvelope{}
err := decodeJSON(r, &request)
if err != nil {
@ -109,16 +108,16 @@ func postLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx
return postLayerRoute, http.StatusBadRequest
}
err = worker.Process(ctx.Store, request.Layer.Format, request.Layer.Name, request.Layer.ParentName, request.Layer.Path, request.Layer.Headers)
err = clair.ProcessLayer(ctx.Store, request.Layer.Format, request.Layer.Name, request.Layer.ParentName, request.Layer.Path, request.Layer.Headers)
if err != nil {
if err == utils.ErrCouldNotExtract ||
err == utils.ErrExtractedFileTooBig ||
err == worker.ErrUnsupported {
if err == tarutil.ErrCouldNotExtract ||
err == tarutil.ErrExtractedFileTooBig ||
err == clair.ErrUnsupported {
writeResponse(w, r, statusUnprocessableEntity, LayerEnvelope{Error: &Error{err.Error()}})
return postLayerRoute, statusUnprocessableEntity
}
if _, badreq := err.(*cerrors.ErrBadRequest); badreq {
if _, badreq := err.(*commonerr.ErrBadRequest); badreq {
writeResponse(w, r, http.StatusBadRequest, LayerEnvelope{Error: &Error{err.Error()}})
return postLayerRoute, http.StatusBadRequest
}
@ -133,17 +132,17 @@ func postLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx
Path: request.Layer.Path,
Headers: request.Layer.Headers,
Format: request.Layer.Format,
IndexedByVersion: worker.Version,
IndexedByVersion: clair.Version,
}})
return postLayerRoute, http.StatusCreated
}
func getLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func getLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
_, withFeatures := r.URL.Query()["features"]
_, withVulnerabilities := r.URL.Query()["vulnerabilities"]
dbLayer, err := ctx.Store.FindLayer(p.ByName("layerName"), withFeatures, withVulnerabilities)
if err == cerrors.ErrNotFound {
if err == commonerr.ErrNotFound {
writeResponse(w, r, http.StatusNotFound, LayerEnvelope{Error: &Error{err.Error()}})
return getLayerRoute, http.StatusNotFound
} else if err != nil {
@ -157,9 +156,9 @@ func getLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *
return getLayerRoute, http.StatusOK
}
func deleteLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func deleteLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
err := ctx.Store.DeleteLayer(p.ByName("layerName"))
if err == cerrors.ErrNotFound {
if err == commonerr.ErrNotFound {
writeResponse(w, r, http.StatusNotFound, LayerEnvelope{Error: &Error{err.Error()}})
return deleteLayerRoute, http.StatusNotFound
} else if err != nil {
@ -171,7 +170,7 @@ func deleteLayer(w http.ResponseWriter, r *http.Request, p httprouter.Params, ct
return deleteLayerRoute, http.StatusOK
}
func getNamespaces(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func getNamespaces(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
dbNamespaces, err := ctx.Store.ListNamespaces()
if err != nil {
writeResponse(w, r, http.StatusInternalServerError, NamespaceEnvelope{Error: &Error{err.Error()}})
@ -189,7 +188,7 @@ func getNamespaces(w http.ResponseWriter, r *http.Request, p httprouter.Params,
return getNamespacesRoute, http.StatusOK
}
func getVulnerabilities(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func getVulnerabilities(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
query := r.URL.Query()
limitStrs, limitExists := query["limit"]
@ -209,7 +208,7 @@ func getVulnerabilities(w http.ResponseWriter, r *http.Request, p httprouter.Par
page := 0
pageStrs, pageExists := query["page"]
if pageExists {
err = tokenUnmarshal(pageStrs[0], ctx.Config.PaginationKey, &page)
err = tokenUnmarshal(pageStrs[0], ctx.PaginationKey, &page)
if err != nil {
writeResponse(w, r, http.StatusBadRequest, VulnerabilityEnvelope{Error: &Error{"invalid page format: " + err.Error()}})
return getNotificationRoute, http.StatusBadRequest
@ -223,7 +222,7 @@ func getVulnerabilities(w http.ResponseWriter, r *http.Request, p httprouter.Par
}
dbVulns, nextPage, err := ctx.Store.ListVulnerabilities(namespace, limit, page)
if err == cerrors.ErrNotFound {
if err == commonerr.ErrNotFound {
writeResponse(w, r, http.StatusNotFound, VulnerabilityEnvelope{Error: &Error{err.Error()}})
return getVulnerabilityRoute, http.StatusNotFound
} else if err != nil {
@ -239,7 +238,7 @@ func getVulnerabilities(w http.ResponseWriter, r *http.Request, p httprouter.Par
var nextPageStr string
if nextPage != -1 {
nextPageBytes, err := tokenMarshal(nextPage, ctx.Config.PaginationKey)
nextPageBytes, err := tokenMarshal(nextPage, ctx.PaginationKey)
if err != nil {
writeResponse(w, r, http.StatusBadRequest, VulnerabilityEnvelope{Error: &Error{"failed to marshal token: " + err.Error()}})
return getNotificationRoute, http.StatusBadRequest
@ -251,7 +250,7 @@ func getVulnerabilities(w http.ResponseWriter, r *http.Request, p httprouter.Par
return getVulnerabilitiesRoute, http.StatusOK
}
func postVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func postVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
request := VulnerabilityEnvelope{}
err := decodeJSON(r, &request)
if err != nil {
@ -273,7 +272,7 @@ func postVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Para
err = ctx.Store.InsertVulnerabilities([]database.Vulnerability{vuln}, true)
if err != nil {
switch err.(type) {
case *cerrors.ErrBadRequest:
case *commonerr.ErrBadRequest:
writeResponse(w, r, http.StatusBadRequest, VulnerabilityEnvelope{Error: &Error{err.Error()}})
return postVulnerabilityRoute, http.StatusBadRequest
default:
@ -286,11 +285,11 @@ func postVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Para
return postVulnerabilityRoute, http.StatusCreated
}
func getVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func getVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
_, withFixedIn := r.URL.Query()["fixedIn"]
dbVuln, err := ctx.Store.FindVulnerability(p.ByName("namespaceName"), p.ByName("vulnerabilityName"))
if err == cerrors.ErrNotFound {
if err == commonerr.ErrNotFound {
writeResponse(w, r, http.StatusNotFound, VulnerabilityEnvelope{Error: &Error{err.Error()}})
return getVulnerabilityRoute, http.StatusNotFound
} else if err != nil {
@ -304,7 +303,7 @@ func getVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Param
return getVulnerabilityRoute, http.StatusOK
}
func putVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func putVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
request := VulnerabilityEnvelope{}
err := decodeJSON(r, &request)
if err != nil {
@ -334,7 +333,7 @@ func putVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Param
err = ctx.Store.InsertVulnerabilities([]database.Vulnerability{vuln}, true)
if err != nil {
switch err.(type) {
case *cerrors.ErrBadRequest:
case *commonerr.ErrBadRequest:
writeResponse(w, r, http.StatusBadRequest, VulnerabilityEnvelope{Error: &Error{err.Error()}})
return putVulnerabilityRoute, http.StatusBadRequest
default:
@ -347,9 +346,9 @@ func putVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Param
return putVulnerabilityRoute, http.StatusOK
}
func deleteVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func deleteVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
err := ctx.Store.DeleteVulnerability(p.ByName("namespaceName"), p.ByName("vulnerabilityName"))
if err == cerrors.ErrNotFound {
if err == commonerr.ErrNotFound {
writeResponse(w, r, http.StatusNotFound, VulnerabilityEnvelope{Error: &Error{err.Error()}})
return deleteVulnerabilityRoute, http.StatusNotFound
} else if err != nil {
@ -361,9 +360,9 @@ func deleteVulnerability(w http.ResponseWriter, r *http.Request, p httprouter.Pa
return deleteVulnerabilityRoute, http.StatusOK
}
func getFixes(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func getFixes(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
dbVuln, err := ctx.Store.FindVulnerability(p.ByName("namespaceName"), p.ByName("vulnerabilityName"))
if err == cerrors.ErrNotFound {
if err == commonerr.ErrNotFound {
writeResponse(w, r, http.StatusNotFound, FeatureEnvelope{Error: &Error{err.Error()}})
return getFixesRoute, http.StatusNotFound
} else if err != nil {
@ -376,7 +375,7 @@ func getFixes(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *
return getFixesRoute, http.StatusOK
}
func putFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func putFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
request := FeatureEnvelope{}
err := decodeJSON(r, &request)
if err != nil {
@ -403,11 +402,11 @@ func putFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *co
err = ctx.Store.InsertVulnerabilityFixes(p.ByName("vulnerabilityNamespace"), p.ByName("vulnerabilityName"), []database.FeatureVersion{dbFix})
if err != nil {
switch err.(type) {
case *cerrors.ErrBadRequest:
case *commonerr.ErrBadRequest:
writeResponse(w, r, http.StatusBadRequest, FeatureEnvelope{Error: &Error{err.Error()}})
return putFixRoute, http.StatusBadRequest
default:
if err == cerrors.ErrNotFound {
if err == commonerr.ErrNotFound {
writeResponse(w, r, http.StatusNotFound, FeatureEnvelope{Error: &Error{err.Error()}})
return putFixRoute, http.StatusNotFound
}
@ -420,9 +419,9 @@ func putFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *co
return putFixRoute, http.StatusOK
}
func deleteFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func deleteFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
err := ctx.Store.DeleteVulnerabilityFix(p.ByName("vulnerabilityNamespace"), p.ByName("vulnerabilityName"), p.ByName("fixName"))
if err == cerrors.ErrNotFound {
if err == commonerr.ErrNotFound {
writeResponse(w, r, http.StatusNotFound, FeatureEnvelope{Error: &Error{err.Error()}})
return deleteFixRoute, http.StatusNotFound
} else if err != nil {
@ -434,7 +433,7 @@ func deleteFix(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx
return deleteFixRoute, http.StatusOK
}
func getNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func getNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
query := r.URL.Query()
limitStrs, limitExists := query["limit"]
@ -452,14 +451,14 @@ func getNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params
page := database.VulnerabilityNotificationFirstPage
pageStrs, pageExists := query["page"]
if pageExists {
err := tokenUnmarshal(pageStrs[0], ctx.Config.PaginationKey, &page)
err := tokenUnmarshal(pageStrs[0], ctx.PaginationKey, &page)
if err != nil {
writeResponse(w, r, http.StatusBadRequest, NotificationEnvelope{Error: &Error{"invalid page format: " + err.Error()}})
return getNotificationRoute, http.StatusBadRequest
}
pageToken = pageStrs[0]
} else {
pageTokenBytes, err := tokenMarshal(page, ctx.Config.PaginationKey)
pageTokenBytes, err := tokenMarshal(page, ctx.PaginationKey)
if err != nil {
writeResponse(w, r, http.StatusBadRequest, NotificationEnvelope{Error: &Error{"failed to marshal token: " + err.Error()}})
return getNotificationRoute, http.StatusBadRequest
@ -468,7 +467,7 @@ func getNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params
}
dbNotification, nextPage, err := ctx.Store.GetNotification(p.ByName("notificationName"), limit, page)
if err == cerrors.ErrNotFound {
if err == commonerr.ErrNotFound {
writeResponse(w, r, http.StatusNotFound, NotificationEnvelope{Error: &Error{err.Error()}})
return deleteNotificationRoute, http.StatusNotFound
} else if err != nil {
@ -476,15 +475,15 @@ func getNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params
return getNotificationRoute, http.StatusInternalServerError
}
notification := NotificationFromDatabaseModel(dbNotification, limit, pageToken, nextPage, ctx.Config.PaginationKey)
notification := NotificationFromDatabaseModel(dbNotification, limit, pageToken, nextPage, ctx.PaginationKey)
writeResponse(w, r, http.StatusOK, NotificationEnvelope{Notification: &notification})
return getNotificationRoute, http.StatusOK
}
func deleteNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func deleteNotification(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
err := ctx.Store.DeleteNotification(p.ByName("notificationName"))
if err == cerrors.ErrNotFound {
if err == commonerr.ErrNotFound {
writeResponse(w, r, http.StatusNotFound, NotificationEnvelope{Error: &Error{err.Error()}})
return deleteNotificationRoute, http.StatusNotFound
} else if err != nil {
@ -496,7 +495,7 @@ func deleteNotification(w http.ResponseWriter, r *http.Request, p httprouter.Par
return deleteNotificationRoute, http.StatusOK
}
func getMetrics(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context.RouteContext) (string, int) {
func getMetrics(w http.ResponseWriter, r *http.Request, p httprouter.Params, ctx *context) (string, int) {
prometheus.Handler().ServeHTTP(w, r)
return getMetricsRoute, 0
}

75
clair.go
View File

@ -1,75 +0,0 @@
// Copyright 2015 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package clair implements the ability to boot Clair with your own imports
// that can dynamically register additional functionality.
package clair
import (
"math/rand"
"os"
"os/signal"
"syscall"
"time"
"github.com/coreos/clair/api"
"github.com/coreos/clair/api/context"
"github.com/coreos/clair/config"
"github.com/coreos/clair/database"
"github.com/coreos/clair/notifier"
"github.com/coreos/clair/updater"
"github.com/coreos/clair/utils"
"github.com/coreos/pkg/capnslog"
)
var log = capnslog.NewPackageLogger("github.com/coreos/clair", "main")
// Boot starts Clair. By exporting this function, anyone can import their own
// custom fetchers/updaters into their own package and then call clair.Boot.
func Boot(config *config.Config) {
rand.Seed(time.Now().UnixNano())
st := utils.NewStopper()
// Open database
db, err := database.Open(config.Database)
if err != nil {
log.Fatal(err)
}
defer db.Close()
// Start notifier
st.Begin()
go notifier.Run(config.Notifier, db, st)
// Start API
st.Begin()
go api.Run(config.API, &context.RouteContext{db, config.API}, st)
st.Begin()
go api.RunHealth(config.API, &context.RouteContext{db, config.API}, st)
// Start updater
st.Begin()
go updater.Run(config.Updater, db, st)
// Wait for interruption and shutdown gracefully.
waitForSignals(syscall.SIGINT, syscall.SIGTERM)
log.Info("Received interruption, gracefully stopping ...")
st.Stop()
}
func waitForSignals(signals ...os.Signal) {
interrupts := make(chan os.Signal, 1)
signal.Notify(interrupts, signals...)
<-interrupts
}

64
config/config.go → cmd/clair/config.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
package config
package main
import (
"errors"
@ -20,21 +20,19 @@ import (
"os"
"time"
"github.com/fernet/fernet-go"
"gopkg.in/yaml.v2"
"github.com/coreos/clair"
"github.com/coreos/clair/api"
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/notification"
"github.com/fernet/fernet-go"
)
// ErrDatasourceNotLoaded is returned when the datasource variable in the configuration file is not loaded properly
// ErrDatasourceNotLoaded is returned when the datasource variable in the
// configuration file is not loaded properly
var ErrDatasourceNotLoaded = errors.New("could not load configuration: no database source specified")
// RegistrableComponentConfig is a configuration block that can be used to
// determine which registrable component should be initialized and pass
// custom configuration to it.
type RegistrableComponentConfig struct {
Type string
Options map[string]interface{}
}
// File represents a YAML configuration file that namespaces all Clair
// configuration under the top-level "clair" key.
type File struct {
@ -43,57 +41,37 @@ type File struct {
// Config is the global configuration for an instance of Clair.
type Config struct {
Database RegistrableComponentConfig
Updater *UpdaterConfig
Notifier *NotifierConfig
API *APIConfig
}
// UpdaterConfig is the configuration for the Updater service.
type UpdaterConfig struct {
Interval time.Duration
}
// NotifierConfig is the configuration for the Notifier service and its registered notifiers.
type NotifierConfig struct {
Attempts int
RenotifyInterval time.Duration
Params map[string]interface{} `yaml:",inline"`
}
// APIConfig is the configuration for the API service.
type APIConfig struct {
Port int
HealthPort int
Timeout time.Duration
PaginationKey string
CertFile, KeyFile, CAFile string
Database database.RegistrableComponentConfig
Updater *clair.UpdaterConfig
Notifier *notification.Config
API *api.Config
}
// DefaultConfig is a configuration that can be used as a fallback value.
func DefaultConfig() Config {
return Config{
Database: RegistrableComponentConfig{
Database: database.RegistrableComponentConfig{
Type: "pgsql",
},
Updater: &UpdaterConfig{
Updater: &clair.UpdaterConfig{
Interval: 1 * time.Hour,
},
API: &APIConfig{
API: &api.Config{
Port: 6060,
HealthPort: 6061,
Timeout: 900 * time.Second,
},
Notifier: &NotifierConfig{
Notifier: &notification.Config{
Attempts: 5,
RenotifyInterval: 2 * time.Hour,
},
}
}
// Load is a shortcut to open a file, read it, and generate a Config.
// LoadConfig is a shortcut to open a file, read it, and generate a Config.
//
// It supports relative and absolute paths. Given "", it returns DefaultConfig.
func Load(path string) (config *Config, err error) {
func LoadConfig(path string) (config *Config, err error) {
var cfgFile File
cfgFile.Clair = DefaultConfig()
if path == "" {

149
cmd/clair/main.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -16,67 +16,51 @@ package main
import (
"flag"
"math/rand"
"os"
"os/exec"
"os/signal"
"runtime/pprof"
"strings"
"syscall"
"time"
"github.com/coreos/pkg/capnslog"
"github.com/coreos/clair"
"github.com/coreos/clair/config"
// Register components
_ "github.com/coreos/clair/notifier/notifiers"
_ "github.com/coreos/clair/updater/fetchers/alpine"
_ "github.com/coreos/clair/updater/fetchers/debian"
_ "github.com/coreos/clair/updater/fetchers/oracle"
_ "github.com/coreos/clair/updater/fetchers/rhel"
_ "github.com/coreos/clair/updater/fetchers/ubuntu"
_ "github.com/coreos/clair/updater/metadata_fetchers/nvd"
_ "github.com/coreos/clair/worker/detectors/data/aci"
_ "github.com/coreos/clair/worker/detectors/data/docker"
_ "github.com/coreos/clair/worker/detectors/feature/apk"
_ "github.com/coreos/clair/worker/detectors/feature/dpkg"
_ "github.com/coreos/clair/worker/detectors/feature/rpm"
_ "github.com/coreos/clair/worker/detectors/namespace/alpinerelease"
_ "github.com/coreos/clair/worker/detectors/namespace/aptsources"
_ "github.com/coreos/clair/worker/detectors/namespace/lsbrelease"
_ "github.com/coreos/clair/worker/detectors/namespace/osrelease"
_ "github.com/coreos/clair/worker/detectors/namespace/redhatrelease"
"github.com/coreos/clair/api"
"github.com/coreos/clair/database"
"github.com/coreos/clair/pkg/stopper"
// Register database driver.
_ "github.com/coreos/clair/database/pgsql"
// Register extensions.
_ "github.com/coreos/clair/ext/featurefmt/apk"
_ "github.com/coreos/clair/ext/featurefmt/dpkg"
_ "github.com/coreos/clair/ext/featurefmt/rpm"
_ "github.com/coreos/clair/ext/featurens/alpinerelease"
_ "github.com/coreos/clair/ext/featurens/aptsources"
_ "github.com/coreos/clair/ext/featurens/lsbrelease"
_ "github.com/coreos/clair/ext/featurens/osrelease"
_ "github.com/coreos/clair/ext/featurens/redhatrelease"
_ "github.com/coreos/clair/ext/imagefmt/aci"
_ "github.com/coreos/clair/ext/imagefmt/docker"
_ "github.com/coreos/clair/ext/notification/webhook"
_ "github.com/coreos/clair/ext/vulnmdsrc/nvd"
_ "github.com/coreos/clair/ext/vulnsrc/alpine"
_ "github.com/coreos/clair/ext/vulnsrc/debian"
_ "github.com/coreos/clair/ext/vulnsrc/oracle"
_ "github.com/coreos/clair/ext/vulnsrc/rhel"
_ "github.com/coreos/clair/ext/vulnsrc/ubuntu"
)
var log = capnslog.NewPackageLogger("github.com/coreos/clair/cmd/clair", "main")
func main() {
// Parse command-line arguments
flag.CommandLine = flag.NewFlagSet(os.Args[0], flag.ExitOnError)
flagConfigPath := flag.String("config", "/etc/clair/config.yaml", "Load configuration from the specified file.")
flagCPUProfilePath := flag.String("cpu-profile", "", "Write a CPU profile to the specified file before exiting.")
flagLogLevel := flag.String("log-level", "info", "Define the logging level.")
flag.Parse()
// Load configuration
config, err := config.Load(*flagConfigPath)
if err != nil {
log.Fatalf("failed to load configuration: %s", err)
}
// Initialize logging system
logLevel, err := capnslog.ParseLevel(strings.ToUpper(*flagLogLevel))
capnslog.SetGlobalLogLevel(logLevel)
capnslog.SetFormatter(capnslog.NewPrettyFormatter(os.Stdout, false))
// Enable CPU Profiling if specified
if *flagCPUProfilePath != "" {
defer stopCPUProfiling(startCPUProfiling(*flagCPUProfilePath))
}
clair.Boot(config)
func waitForSignals(signals ...os.Signal) {
interrupts := make(chan os.Signal, 1)
signal.Notify(interrupts, signals...)
<-interrupts
}
func startCPUProfiling(path string) *os.File {
@ -100,3 +84,70 @@ func stopCPUProfiling(f *os.File) {
f.Close()
log.Info("stopped CPU profiling")
}
// Boot starts Clair instance with the provided config.
func Boot(config *Config) {
rand.Seed(time.Now().UnixNano())
st := stopper.NewStopper()
// Open database
db, err := database.Open(config.Database)
if err != nil {
log.Fatal(err)
}
defer db.Close()
// Start notifier
st.Begin()
go clair.RunNotifier(config.Notifier, db, st)
// Start API
st.Begin()
go api.Run(config.API, db, st)
st.Begin()
go api.RunHealth(config.API, db, st)
// Start updater
st.Begin()
go clair.RunUpdater(config.Updater, db, st)
// Wait for interruption and shutdown gracefully.
waitForSignals(syscall.SIGINT, syscall.SIGTERM)
log.Info("Received interruption, gracefully stopping ...")
st.Stop()
}
func main() {
// Parse command-line arguments
flag.CommandLine = flag.NewFlagSet(os.Args[0], flag.ExitOnError)
flagConfigPath := flag.String("config", "/etc/clair/config.yaml", "Load configuration from the specified file.")
flagCPUProfilePath := flag.String("cpu-profile", "", "Write a CPU profile to the specified file before exiting.")
flagLogLevel := flag.String("log-level", "info", "Define the logging level.")
flag.Parse()
// Check for dependencies.
for _, bin := range []string{"git", "bzr", "rpm", "xz"} {
_, err := exec.LookPath(bin)
if err != nil {
log.Fatalf("failed to find dependency: %s", bin)
}
}
// Load configuration
config, err := LoadConfig(*flagConfigPath)
if err != nil {
log.Fatalf("failed to load configuration: %s", err)
}
// Initialize logging system
logLevel, err := capnslog.ParseLevel(strings.ToUpper(*flagLogLevel))
capnslog.SetGlobalLogLevel(logLevel)
capnslog.SetFormatter(capnslog.NewPrettyFormatter(os.Stdout, false))
// Enable CPU Profiling if specified
if *flagCPUProfilePath != "" {
defer stopCPUProfiling(startCPUProfiling(*flagCPUProfilePath))
}
Boot(config)
}

183
database/database.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,15 +12,14 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package database defines the Clair's models and a common interface for database implementations.
// Package database defines the Clair's models and a common interface for
// database implementations.
package database
import (
"errors"
"fmt"
"time"
"github.com/coreos/clair/config"
)
var (
@ -29,15 +28,24 @@ var (
ErrBackendException = errors.New("database: an error occured when querying the backend")
// ErrInconsistent is an error that occurs when a database consistency check
// fails (ie. when an entity which is supposed to be unique is detected twice)
// fails (i.e. when an entity which is supposed to be unique is detected
// twice)
ErrInconsistent = errors.New("database: inconsistent database")
)
// RegistrableComponentConfig is a configuration block that can be used to
// determine which registrable component should be initialized and pass custom
// configuration to it.
type RegistrableComponentConfig struct {
Type string
Options map[string]interface{}
}
var drivers = make(map[string]Driver)
// Driver is a function that opens a Datastore specified by its database driver type and specific
// configuration.
type Driver func(config.RegistrableComponentConfig) (Datastore, error)
type Driver func(RegistrableComponentConfig) (Datastore, error)
// Register makes a Constructor available by the provided name.
//
@ -54,7 +62,7 @@ func Register(name string, driver Driver) {
}
// Open opens a Datastore specified by a configuration.
func Open(cfg config.RegistrableComponentConfig) (Datastore, error) {
func Open(cfg RegistrableComponentConfig) (Datastore, error) {
driver, ok := drivers[cfg.Type]
if !ok {
return nil, fmt.Errorf("database: unknown Driver %q (forgotten configuration or import?)", cfg.Type)
@ -62,123 +70,152 @@ func Open(cfg config.RegistrableComponentConfig) (Datastore, error) {
return driver(cfg)
}
// Datastore is the interface that describes a database backend implementation.
// Datastore represents the required operations on a persistent data store for
// a Clair deployment.
type Datastore interface {
// # Namespace
// ListNamespaces returns the entire list of known Namespaces.
ListNamespaces() ([]Namespace, error)
// # Layer
// InsertLayer stores a Layer in the database.
// A Layer is uniquely identified by its Name. The Name and EngineVersion fields are mandatory.
// If a Parent is specified, it is expected that it has been retrieved using FindLayer.
// If a Layer that already exists is inserted and the EngineVersion of the given Layer is higher
// than the stored one, the stored Layer should be updated.
// The function has to be idempotent, inserting a layer that already exists shouln'd return an
// error.
//
// A Layer is uniquely identified by its Name.
// The Name and EngineVersion fields are mandatory.
// If a Parent is specified, it is expected that it has been retrieved using
// FindLayer.
// If a Layer that already exists is inserted and the EngineVersion of the
// given Layer is higher than the stored one, the stored Layer should be
// updated.
// The function has to be idempotent, inserting a layer that already exists
// shouldn't return an error.
InsertLayer(Layer) error
// FindLayer retrieves a Layer from the database.
// withFeatures specifies whether the Features field should be filled. When withVulnerabilities is
// true, the Features field should be filled and their AffectedBy fields should contain every
// vulnerabilities that affect them.
//
// When `withFeatures` is true, the Features field should be filled.
// When `withVulnerabilities` is true, the Features field should be filled
// and their AffectedBy fields should contain every vulnerabilities that
// affect them.
FindLayer(name string, withFeatures, withVulnerabilities bool) (Layer, error)
// DeleteLayer deletes a Layer from the database and every layers that are based on it,
// recursively.
// DeleteLayer deletes a Layer from the database and every layers that are
// based on it, recursively.
DeleteLayer(name string) error
// # Vulnerability
// ListVulnerabilities returns the list of vulnerabilies of a certain Namespace.
// ListVulnerabilities returns the list of vulnerabilities of a particular
// Namespace.
//
// The Limit and page parameters are used to paginate the return list.
// The first given page should be 0. The function will then return the next available page.
// If there is no more page, -1 has to be returned.
// The first given page should be 0.
// The function should return the next available page. If there are no more
// pages, -1 has to be returned.
ListVulnerabilities(namespaceName string, limit int, page int) ([]Vulnerability, int, error)
// InsertVulnerabilities stores the given Vulnerabilities in the database, updating them if
// necessary. A vulnerability is uniquely identified by its Namespace and its Name.
// The FixedIn field may only contain a partial list of Features that are affected by the
// Vulnerability, along with the version in which the vulnerability is fixed. It is the
// responsibility of the implementation to update the list properly. A version equals to
// types.MinVersion means that the given Feature is not being affected by the Vulnerability at
// all and thus, should be removed from the list. It is important that Features should be unique
// in the FixedIn list. For example, it doesn't make sense to have two `openssl` Feature listed as
// a Vulnerability can only be fixed in one Version. This is true because Vulnerabilities and
// Features are Namespaced (i.e. specific to one operating system).
// Each vulnerability insertion or update has to create a Notification that will contain the
// old and the updated Vulnerability, unless createNotification equals to true.
// InsertVulnerabilities stores the given Vulnerabilities in the database,
// updating them if necessary.
//
// A vulnerability is uniquely identified by its Namespace and its Name.
// The FixedIn field may only contain a partial list of Features that are
// affected by the Vulnerability, along with the version in which the
// vulnerability is fixed. It is the responsibility of the implementation to
// update the list properly.
// A version equals to versionfmt.MinVersion means that the given Feature is
// not being affected by the Vulnerability at all and thus, should be removed
// from the list.
// It is important that Features should be unique in the FixedIn list. For
// example, it doesn't make sense to have two `openssl` Feature listed as a
// Vulnerability can only be fixed in one Version. This is true because
// Vulnerabilities and Features are namespaced (i.e. specific to one
// operating system).
// Each vulnerability insertion or update has to create a Notification that
// will contain the old and the updated Vulnerability, unless
// createNotification equals to true.
InsertVulnerabilities(vulnerabilities []Vulnerability, createNotification bool) error
// FindVulnerability retrieves a Vulnerability from the database, including the FixedIn list.
// FindVulnerability retrieves a Vulnerability from the database, including
// the FixedIn list.
FindVulnerability(namespaceName, name string) (Vulnerability, error)
// DeleteVulnerability removes a Vulnerability from the database.
//
// It has to create a Notification that will contain the old Vulnerability.
DeleteVulnerability(namespaceName, name string) error
// InsertVulnerabilityFixes adds new FixedIn Feature or update the Versions of existing ones to
// the specified Vulnerability in the database.
// It has has to create a Notification that will contain the old and the updated Vulnerability.
// InsertVulnerabilityFixes adds new FixedIn Feature or update the Versions
// of existing ones to the specified Vulnerability in the database.
//
// It has has to create a Notification that will contain the old and the
// updated Vulnerability.
InsertVulnerabilityFixes(vulnerabilityNamespace, vulnerabilityName string, fixes []FeatureVersion) error
// DeleteVulnerabilityFix removes a FixedIn Feature from the specified Vulnerability in the
// database. It can be used to store the fact that a Vulnerability no longer affects the given
// Feature in any Version.
// It has has to create a Notification that will contain the old and the updated Vulnerability.
// DeleteVulnerabilityFix removes a FixedIn Feature from the specified
// Vulnerability in the database. It can be used to store the fact that a
// Vulnerability no longer affects the given Feature in any Version.
//
// It has has to create a Notification that will contain the old and the
// updated Vulnerability.
DeleteVulnerabilityFix(vulnerabilityNamespace, vulnerabilityName, featureName string) error
// # Notification
// GetAvailableNotification returns the Name, Created, Notified and Deleted fields of a
// Notification that should be handled. The renotify interval defines how much time after being
// marked as Notified by SetNotificationNotified, a Notification that hasn't been deleted should
// be returned again by this function. A Notification for which there is a valid Lock with the
// same Name should not be returned.
// GetAvailableNotification returns the Name, Created, Notified and Deleted
// fields of a Notification that should be handled.
//
// The renotify interval defines how much time after being marked as Notified
// by SetNotificationNotified, a Notification that hasn't been deleted should
// be returned again by this function.
// A Notification for which there is a valid Lock with the same Name should
// not be returned.
GetAvailableNotification(renotifyInterval time.Duration) (VulnerabilityNotification, error)
// GetNotification returns a Notification, including its OldVulnerability and NewVulnerability
// fields. On these Vulnerabilities, LayersIntroducingVulnerability should be filled with
// every Layer that introduces the Vulnerability (i.e. adds at least one affected FeatureVersion).
// The Limit and page parameters are used to paginate LayersIntroducingVulnerability. The first
// given page should be VulnerabilityNotificationFirstPage. The function will then return the next
// availage page. If there is no more page, NoVulnerabilityNotificationPage has to be returned.
// GetNotification returns a Notification, including its OldVulnerability and
// NewVulnerability fields.
//
// On these Vulnerabilities, LayersIntroducingVulnerability should be filled
// with every Layer that introduces the Vulnerability (i.e. adds at least one
// affected FeatureVersion).
// The Limit and page parameters are used to paginate
// LayersIntroducingVulnerability. The first given page should be
// VulnerabilityNotificationFirstPage. The function will then return the next
// available page. If there is no more page, NoVulnerabilityNotificationPage
// has to be returned.
GetNotification(name string, limit int, page VulnerabilityNotificationPageNumber) (VulnerabilityNotification, VulnerabilityNotificationPageNumber, error)
// SetNotificationNotified marks a Notification as notified and thus, makes it unavailable for
// GetAvailableNotification, until the renotify duration is elapsed.
// SetNotificationNotified marks a Notification as notified and thus, makes
// it unavailable for GetAvailableNotification, until the renotify duration
// is elapsed.
SetNotificationNotified(name string) error
// DeleteNotification marks a Notification as deleted, and thus, makes it unavailable for
// GetAvailableNotification.
// DeleteNotification marks a Notification as deleted, and thus, makes it
// unavailable for GetAvailableNotification.
DeleteNotification(name string) error
// # Key/Value
// InsertKeyValue stores or updates a simple key/value pair in the database.
InsertKeyValue(key, value string) error
// GetKeyValue retrieves a value from the database from the given key.
//
// It returns an empty string if there is no such key.
GetKeyValue(key string) (string, error)
// # Lock
// Lock creates or renew a Lock in the database with the given name, owner and duration.
// After the specified duration, the Lock expires by itself if it hasn't been unlocked, and thus,
// let other users create a Lock with the same name. However, the owner can renew its Lock by
// setting renew to true. Lock should not block, it should instead returns whether the Lock has
// been successfully acquired/renewed. If it's the case, the expiration time of that Lock is
// returned as well.
// Lock creates or renew a Lock in the database with the given name, owner
// and duration.
//
// After the specified duration, the Lock expires by itself if it hasn't been
// unlocked, and thus, let other users create a Lock with the same name.
// However, the owner can renew its Lock by setting renew to true.
// Lock should not block, it should instead returns whether the Lock has been
// successfully acquired/renewed. If it's the case, the expiration time of
// that Lock is returned as well.
Lock(name string, owner string, duration time.Duration, renew bool) (bool, time.Time)
// Unlock releases an existing Lock.
Unlock(name, owner string)
// FindLock returns the owner of a Lock specified by the name, and its experation time if it
// exists.
// FindLock returns the owner of a Lock specified by the name, and its
// expiration time if it exists.
FindLock(name string) (string, time.Time, error)
// # Miscellaneous
// Ping returns the health status of the database.
Ping() bool
// Close closes the database and free any allocated resource.
// Close closes the database and frees any allocated resource.
Close()
}

6
database/models.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -18,8 +18,6 @@ import (
"database/sql/driver"
"encoding/json"
"time"
"github.com/coreos/clair/utils/types"
)
// ID is only meant to be used by database implementations and should never be used for anything else.
@ -70,7 +68,7 @@ type Vulnerability struct {
Description string
Link string
Severity types.Priority
Severity Severity
Metadata MetadataMap

10
database/pgsql/complex_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -28,8 +28,6 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt/dpkg"
"github.com/coreos/clair/utils"
"github.com/coreos/clair/utils/types"
)
const (
@ -93,7 +91,7 @@ func TestRaceAffects(t *testing.T) {
Version: strconv.Itoa(version),
},
},
Severity: types.Unknown,
Severity: database.UnknownSeverity,
}
vulnerabilities[version] = append(vulnerabilities[version], vulnerability)
@ -157,7 +155,7 @@ func TestRaceAffects(t *testing.T) {
}
}
assert.Len(t, utils.CompareStringLists(expectedAffectedNames, actualAffectedNames), 0)
assert.Len(t, utils.CompareStringLists(actualAffectedNames, expectedAffectedNames), 0)
assert.Len(t, compareStringLists(expectedAffectedNames, actualAffectedNames), 0)
assert.Len(t, compareStringLists(actualAffectedNames, expectedAffectedNames), 0)
}
}

8
database/pgsql/feature.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -21,12 +21,12 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/pkg/commonerr"
)
func (pgSQL *pgSQL) insertFeature(feature database.Feature) (int, error) {
if feature.Name == "" {
return 0, cerrors.NewBadRequestError("could not find/insert invalid Feature")
return 0, commonerr.NewBadRequestError("could not find/insert invalid Feature")
}
// Do cache lookup.
@ -65,7 +65,7 @@ func (pgSQL *pgSQL) insertFeature(feature database.Feature) (int, error) {
func (pgSQL *pgSQL) insertFeatureVersion(fv database.FeatureVersion) (id int, err error) {
err = versionfmt.Valid(fv.Feature.Namespace.VersionFormat, fv.Version)
if err != nil {
return 0, cerrors.NewBadRequestError("could not find/insert invalid FeatureVersion")
return 0, commonerr.NewBadRequestError("could not find/insert invalid FeatureVersion")
}
// Do cache lookup.

6
database/pgsql/keyvalue.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -18,14 +18,14 @@ import (
"database/sql"
"time"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/pkg/commonerr"
)
// InsertKeyValue stores (or updates) a single key / value tuple.
func (pgSQL *pgSQL) InsertKeyValue(key, value string) (err error) {
if key == "" || value == "" {
log.Warning("could not insert a flag which has an empty name or value")
return cerrors.NewBadRequestError("could not insert a flag which has an empty name or value")
return commonerr.NewBadRequestError("could not insert a flag which has an empty name or value")
}
defer observeQueryTime("InsertKeyValue", "all", time.Now())

17
database/pgsql/layer.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -22,8 +22,7 @@ import (
"github.com/guregu/null/zero"
"github.com/coreos/clair/database"
"github.com/coreos/clair/utils"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/pkg/commonerr"
)
func (pgSQL *pgSQL) FindLayer(name string, withFeatures, withVulnerabilities bool) (database.Layer, error) {
@ -247,12 +246,12 @@ func (pgSQL *pgSQL) InsertLayer(layer database.Layer) error {
// Verify parameters
if layer.Name == "" {
log.Warning("could not insert a layer which has an empty Name")
return cerrors.NewBadRequestError("could not insert a layer which has an empty Name")
return commonerr.NewBadRequestError("could not insert a layer which has an empty Name")
}
// Get a potentially existing layer.
existingLayer, err := pgSQL.FindLayer(layer.Name, true, false)
if err != nil && err != cerrors.ErrNotFound {
if err != nil && err != commonerr.ErrNotFound {
return err
} else if err == nil {
if existingLayer.EngineVersion >= layer.EngineVersion {
@ -271,7 +270,7 @@ func (pgSQL *pgSQL) InsertLayer(layer database.Layer) error {
if layer.Parent != nil {
if layer.Parent.ID == 0 {
log.Warning("Parent is expected to be retrieved from database when inserting a layer.")
return cerrors.NewBadRequestError("Parent is expected to be retrieved from database when inserting a layer.")
return commonerr.NewBadRequestError("Parent is expected to be retrieved from database when inserting a layer.")
}
parentID = zero.IntFrom(int64(layer.Parent.ID))
@ -362,8 +361,8 @@ func (pgSQL *pgSQL) updateDiffFeatureVersions(tx *sql.Tx, layer, existingLayer *
parentLayerFeaturesMapNV, parentLayerFeaturesNV := createNV(layer.Parent.Features)
// Calculate the added and deleted FeatureVersions name:version.
addNV := utils.CompareStringLists(layerFeaturesNV, parentLayerFeaturesNV)
delNV := utils.CompareStringLists(parentLayerFeaturesNV, layerFeaturesNV)
addNV := compareStringLists(layerFeaturesNV, parentLayerFeaturesNV)
delNV := compareStringLists(parentLayerFeaturesNV, layerFeaturesNV)
// Fill the structures containing the added and deleted FeatureVersions.
for _, nv := range addNV {
@ -429,7 +428,7 @@ func (pgSQL *pgSQL) DeleteLayer(name string) error {
}
if affected <= 0 {
return cerrors.ErrNotFound
return commonerr.ErrNotFound
}
return nil

15
database/pgsql/layer_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -22,8 +22,7 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt/dpkg"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/utils/types"
"github.com/coreos/clair/pkg/commonerr"
)
func TestFindLayer(t *testing.T) {
@ -91,7 +90,7 @@ func TestFindLayer(t *testing.T) {
if assert.Len(t, featureVersion.AffectedBy, 1) {
assert.Equal(t, "debian:7", featureVersion.AffectedBy[0].Namespace.Name)
assert.Equal(t, "CVE-OPENSSL-1-DEB7", featureVersion.AffectedBy[0].Name)
assert.Equal(t, types.High, featureVersion.AffectedBy[0].Severity)
assert.Equal(t, database.HighSeverity, featureVersion.AffectedBy[0].Severity)
assert.Equal(t, "A vulnerability affecting OpenSSL < 2.0 on Debian 7.0", featureVersion.AffectedBy[0].Description)
assert.Equal(t, "http://google.com/#q=CVE-OPENSSL-1-DEB7", featureVersion.AffectedBy[0].Link)
assert.Equal(t, "2.0", featureVersion.AffectedBy[0].FixedBy)
@ -363,19 +362,19 @@ func testInsertLayerUpdate(t *testing.T, datastore database.Datastore) {
func testInsertLayerDelete(t *testing.T, datastore database.Datastore) {
err := datastore.DeleteLayer("TestInsertLayerX")
assert.Equal(t, cerrors.ErrNotFound, err)
assert.Equal(t, commonerr.ErrNotFound, err)
err = datastore.DeleteLayer("TestInsertLayer3")
assert.Nil(t, err)
_, err = datastore.FindLayer("TestInsertLayer3", false, false)
assert.Equal(t, cerrors.ErrNotFound, err)
assert.Equal(t, commonerr.ErrNotFound, err)
_, err = datastore.FindLayer("TestInsertLayer4a", false, false)
assert.Equal(t, cerrors.ErrNotFound, err)
assert.Equal(t, commonerr.ErrNotFound, err)
_, err = datastore.FindLayer("TestInsertLayer4b", true, false)
assert.Equal(t, cerrors.ErrNotFound, err)
assert.Equal(t, commonerr.ErrNotFound, err)
}
func cmpFV(a, b database.FeatureVersion) bool {

6
database/pgsql/lock.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -17,7 +17,7 @@ package pgsql
import (
"time"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/pkg/commonerr"
)
// Lock tries to set a temporary lock in the database.
@ -80,7 +80,7 @@ func (pgSQL *pgSQL) Unlock(name, owner string) {
func (pgSQL *pgSQL) FindLock(name string) (string, time.Time, error) {
if name == "" {
log.Warning("could not find an invalid lock")
return "", time.Time{}, cerrors.NewBadRequestError("could not find an invalid lock")
return "", time.Time{}, commonerr.NewBadRequestError("could not find an invalid lock")
}
defer observeQueryTime("FindLock", "all", time.Now())

6
database/pgsql/namespace.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -18,12 +18,12 @@ import (
"time"
"github.com/coreos/clair/database"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/pkg/commonerr"
)
func (pgSQL *pgSQL) insertNamespace(namespace database.Namespace) (int, error) {
if namespace.Name == "" {
return 0, cerrors.NewBadRequestError("could not find/insert invalid Namespace")
return 0, commonerr.NewBadRequestError("could not find/insert invalid Namespace")
}
if pgSQL.cache != nil {

6
database/pgsql/notification.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -19,7 +19,7 @@ import (
"time"
"github.com/coreos/clair/database"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/pkg/commonerr"
"github.com/guregu/null/zero"
"github.com/pborman/uuid"
)
@ -242,7 +242,7 @@ func (pgSQL *pgSQL) DeleteNotification(name string) error {
}
if affected <= 0 {
return cerrors.ErrNotFound
return commonerr.ErrNotFound
}
return nil

13
database/pgsql/notification_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -23,8 +23,7 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/ext/versionfmt/dpkg"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/utils/types"
"github.com/coreos/clair/pkg/commonerr"
)
func TestNotification(t *testing.T) {
@ -37,7 +36,7 @@ func TestNotification(t *testing.T) {
// Try to get a notification when there is none.
_, err = datastore.GetAvailableNotification(time.Second)
assert.Equal(t, cerrors.ErrNotFound, err)
assert.Equal(t, commonerr.ErrNotFound, err)
// Create some data.
f1 := database.Feature{
@ -126,7 +125,7 @@ func TestNotification(t *testing.T) {
// Verify the renotify behaviour.
if assert.Nil(t, datastore.SetNotificationNotified(notification.Name)) {
_, err := datastore.GetAvailableNotification(time.Second)
assert.Equal(t, cerrors.ErrNotFound, err)
assert.Equal(t, commonerr.ErrNotFound, err)
time.Sleep(50 * time.Millisecond)
notificationB, err := datastore.GetAvailableNotification(20 * time.Millisecond)
@ -164,12 +163,12 @@ func TestNotification(t *testing.T) {
assert.Nil(t, datastore.DeleteNotification(notification.Name))
_, err = datastore.GetAvailableNotification(time.Millisecond)
assert.Equal(t, cerrors.ErrNotFound, err)
assert.Equal(t, commonerr.ErrNotFound, err)
}
// Update a vulnerability and ensure that the old/new vulnerabilities are correct.
v1b := v1
v1b.Severity = types.High
v1b.Severity = database.HighSeverity
v1b.FixedIn = []database.FeatureVersion{
{
Feature: f1,

28
database/pgsql/pgsql.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -31,11 +31,9 @@ import (
"github.com/prometheus/client_golang/prometheus"
"github.com/remind101/migrate"
"github.com/coreos/clair/config"
"github.com/coreos/clair/database"
"github.com/coreos/clair/database/pgsql/migrations"
"github.com/coreos/clair/utils"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/pkg/commonerr"
)
var (
@ -115,11 +113,13 @@ type Config struct {
FixturePath string
}
// openDatabase opens a PostgresSQL-backed Datastore using the given configuration.
// It immediately every necessary migrations. If ManageDatabaseLifecycle is specified,
// the database will be created first. If FixturePath is specified, every SQL queries that are
// present insides will be executed.
func openDatabase(registrableComponentConfig config.RegistrableComponentConfig) (database.Datastore, error) {
// openDatabase opens a PostgresSQL-backed Datastore using the given
// configuration.
//
// It immediately runs all necessary migrations. If ManageDatabaseLifecycle is
// specified, the database will be created first. If FixturePath is specified,
// every SQL queries that are present insides will be executed.
func openDatabase(registrableComponentConfig database.RegistrableComponentConfig) (database.Datastore, error) {
var pg pgSQL
var err error
@ -196,12 +196,12 @@ func openDatabase(registrableComponentConfig config.RegistrableComponentConfig)
func parseConnectionString(source string) (dbName string, pgSourceURL string, err error) {
if source == "" {
return "", "", cerrors.NewBadRequestError("pgsql: no database connection string specified")
return "", "", commonerr.NewBadRequestError("pgsql: no database connection string specified")
}
sourceURL, err := url.Parse(source)
if err != nil {
return "", "", cerrors.NewBadRequestError("pgsql: database connection string is not a valid URL")
return "", "", commonerr.NewBadRequestError("pgsql: database connection string is not a valid URL")
}
dbName = strings.TrimPrefix(sourceURL.Path, "/")
@ -280,7 +280,7 @@ func handleError(desc string, err error) error {
}
if err == sql.ErrNoRows {
return cerrors.ErrNotFound
return commonerr.ErrNotFound
}
log.Errorf("%s: %v", desc, err)
@ -300,5 +300,7 @@ func isErrUniqueViolation(err error) bool {
}
func observeQueryTime(query, subquery string, start time.Time) {
utils.PrometheusObserveTimeMilliseconds(promQueryDurationMilliseconds.WithLabelValues(query, subquery), start)
promQueryDurationMilliseconds.
WithLabelValues(query, subquery).
Observe(float64(time.Since(start).Nanoseconds()) / float64(time.Millisecond))
}

7
database/pgsql/pgsql_test.go
View File

@ -21,8 +21,9 @@ import (
"runtime"
"strings"
"github.com/coreos/clair/config"
"github.com/pborman/uuid"
"github.com/coreos/clair/database"
)
func openDatabaseForTest(testName string, loadFixture bool) (*pgSQL, error) {
@ -34,7 +35,7 @@ func openDatabaseForTest(testName string, loadFixture bool) (*pgSQL, error) {
return datastore, nil
}
func generateTestConfig(testName string, loadFixture bool) config.RegistrableComponentConfig {
func generateTestConfig(testName string, loadFixture bool) database.RegistrableComponentConfig {
dbName := "test_" + strings.ToLower(testName) + "_" + strings.Replace(uuid.New(), "-", "_", -1)
var fixturePath string
@ -48,7 +49,7 @@ func generateTestConfig(testName string, loadFixture bool) config.RegistrableCom
source = fmt.Sprintf(sourceEnv, dbName)
}
return config.RegistrableComponentConfig{
return database.RegistrableComponentConfig{
Options: map[string]interface{}{
"source": source,
"cachesize": 0,

72
database/pgsql/vulnerability.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -17,17 +17,54 @@ package pgsql
import (
"database/sql"
"encoding/json"
"fmt"
"reflect"
"time"
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/utils"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/pkg/commonerr"
"github.com/guregu/null/zero"
)
// compareStringLists returns the strings that are present in X but not in Y.
func compareStringLists(X, Y []string) []string {
m := make(map[string]bool)
for _, y := range Y {
m[y] = true
}
diff := []string{}
for _, x := range X {
if m[x] {
continue
}
diff = append(diff, x)
m[x] = true
}
return diff
}
func compareStringListsInBoth(X, Y []string) []string {
m := make(map[string]struct{})
for _, y := range Y {
m[y] = struct{}{}
}
diff := []string{}
for _, x := range X {
if _, e := m[x]; e {
diff = append(diff, x)
delete(m, x)
}
}
return diff
}
func (pgSQL *pgSQL) ListVulnerabilities(namespaceName string, limit int, startID int) ([]database.Vulnerability, int, error) {
defer observeQueryTime("listVulnerabilities", "all", time.Now())
@ -37,7 +74,7 @@ func (pgSQL *pgSQL) ListVulnerabilities(namespaceName string, limit int, startID
if err != nil {
return nil, -1, handleError("searchNamespace", err)
} else if id == 0 {
return nil, -1, cerrors.ErrNotFound
return nil, -1, commonerr.ErrNotFound
}
// Query.
@ -130,7 +167,7 @@ func scanVulnerability(queryer Queryer, queryName string, vulnerabilityRow *sql.
}
if vulnerability.ID == 0 {
return vulnerability, cerrors.ErrNotFound
return vulnerability, commonerr.ErrNotFound
}
// Query the FixedIn FeatureVersion now.
@ -195,13 +232,9 @@ func (pgSQL *pgSQL) insertVulnerability(vulnerability database.Vulnerability, on
// Verify parameters
if vulnerability.Name == "" || vulnerability.Namespace.Name == "" {
return cerrors.NewBadRequestError("insertVulnerability needs at least the Name and the Namespace")
}
if !onlyFixedIn && !vulnerability.Severity.IsValid() {
msg := fmt.Sprintf("could not insert a vulnerability that has an invalid Severity: %s", vulnerability.Severity)
log.Warning(msg)
return cerrors.NewBadRequestError(msg)
return commonerr.NewBadRequestError("insertVulnerability needs at least the Name and the Namespace")
}
for i := 0; i < len(vulnerability.FixedIn); i++ {
fifv := &vulnerability.FixedIn[i]
@ -212,7 +245,7 @@ func (pgSQL *pgSQL) insertVulnerability(vulnerability database.Vulnerability, on
} else if fifv.Feature.Namespace.Name != vulnerability.Namespace.Name {
msg := "could not insert an invalid vulnerability that contains FixedIn FeatureVersion that are not in the same namespace as the Vulnerability"
log.Warning(msg)
return cerrors.NewBadRequestError(msg)
return commonerr.NewBadRequestError(msg)
}
}
@ -228,7 +261,7 @@ func (pgSQL *pgSQL) insertVulnerability(vulnerability database.Vulnerability, on
// Find existing vulnerability and its Vulnerability_FixedIn_Features (for update).
existingVulnerability, err := findVulnerability(tx, vulnerability.Namespace.Name, vulnerability.Name, true)
if err != nil && err != cerrors.ErrNotFound {
if err != nil && err != commonerr.ErrNotFound {
tx.Rollback()
return err
}
@ -237,7 +270,7 @@ func (pgSQL *pgSQL) insertVulnerability(vulnerability database.Vulnerability, on
// Because this call tries to update FixedIn FeatureVersion, import all other data from the
// existing one.
if existingVulnerability.ID == 0 {
return cerrors.ErrNotFound
return commonerr.ErrNotFound
}
fixedIn := vulnerability.FixedIn
@ -271,8 +304,9 @@ func (pgSQL *pgSQL) insertVulnerability(vulnerability database.Vulnerability, on
return handleError("removeVulnerability", err)
}
} else {
// The vulnerability is new, we don't want to have any types.MinVersion as they are only used
// for diffing existing vulnerabilities.
// The vulnerability is new, we don't want to have any
// versionfmt.MinVersion as they are only used for diffing existing
// vulnerabilities.
var fixedIn []database.FeatureVersion
for _, fv := range vulnerability.FixedIn {
if fv.Version != versionfmt.MinVersion {
@ -345,8 +379,8 @@ func applyFixedInDiff(currentList, diff []database.FeatureVersion) ([]database.F
currentMap, currentNames := createFeatureVersionNameMap(currentList)
diffMap, diffNames := createFeatureVersionNameMap(diff)
addedNames := utils.CompareStringLists(diffNames, currentNames)
inBothNames := utils.CompareStringListsInBoth(diffNames, currentNames)
addedNames := compareStringLists(diffNames, currentNames)
inBothNames := compareStringListsInBoth(diffNames, currentNames)
different := false

46
database/pgsql/vulnerability_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -23,8 +23,7 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/ext/versionfmt/dpkg"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/utils/types"
"github.com/coreos/clair/pkg/commonerr"
)
func TestFindVulnerability(t *testing.T) {
@ -37,14 +36,14 @@ func TestFindVulnerability(t *testing.T) {
// Find a vulnerability that does not exist.
_, err = datastore.FindVulnerability("", "")
assert.Equal(t, cerrors.ErrNotFound, err)
assert.Equal(t, commonerr.ErrNotFound, err)
// Find a normal vulnerability.
v1 := database.Vulnerability{
Name: "CVE-OPENSSL-1-DEB7",
Description: "A vulnerability affecting OpenSSL < 2.0 on Debian 7.0",
Link: "http://google.com/#q=CVE-OPENSSL-1-DEB7",
Severity: types.High,
Severity: database.HighSeverity,
Namespace: database.Namespace{
Name: "debian:7",
VersionFormat: dpkg.ParserName,
@ -74,7 +73,7 @@ func TestFindVulnerability(t *testing.T) {
Name: "debian:7",
VersionFormat: dpkg.ParserName,
},
Severity: types.Unknown,
Severity: database.UnknownSeverity,
}
v2f, err := datastore.FindVulnerability("debian:7", "CVE-NOPE")
@ -93,15 +92,15 @@ func TestDeleteVulnerability(t *testing.T) {
// Delete non-existing Vulnerability.
err = datastore.DeleteVulnerability("TestDeleteVulnerabilityNamespace1", "CVE-OPENSSL-1-DEB7")
assert.Equal(t, cerrors.ErrNotFound, err)
assert.Equal(t, commonerr.ErrNotFound, err)
err = datastore.DeleteVulnerability("debian:7", "TestDeleteVulnerabilityVulnerability1")
assert.Equal(t, cerrors.ErrNotFound, err)
assert.Equal(t, commonerr.ErrNotFound, err)
// Delete Vulnerability.
err = datastore.DeleteVulnerability("debian:7", "CVE-OPENSSL-1-DEB7")
if assert.Nil(t, err) {
_, err := datastore.FindVulnerability("debian:7", "CVE-OPENSSL-1-DEB7")
assert.Equal(t, cerrors.ErrNotFound, err)
assert.Equal(t, commonerr.ErrNotFound, err)
}
}
@ -180,30 +179,24 @@ func TestInsertVulnerability(t *testing.T) {
Name: "",
Namespace: n1,
FixedIn: []database.FeatureVersion{f1},
Severity: types.Unknown,
Severity: database.UnknownSeverity,
},
{
Name: "TestInsertVulnerability0",
Namespace: database.Namespace{},
FixedIn: []database.FeatureVersion{f1},
Severity: types.Unknown,
Severity: database.UnknownSeverity,
},
{
Name: "TestInsertVulnerability0-",
Namespace: database.Namespace{},
FixedIn: []database.FeatureVersion{f1},
},
{
Name: "TestInsertVulnerability0",
Namespace: n1,
FixedIn: []database.FeatureVersion{f1},
Severity: types.Priority(""),
},
{
Name: "TestInsertVulnerability0",
Namespace: n1,
FixedIn: []database.FeatureVersion{f2},
Severity: types.Unknown,
Severity: database.UnknownSeverity,
},
} {
err := datastore.InsertVulnerabilities([]database.Vulnerability{vulnerability}, true)
@ -223,7 +216,7 @@ func TestInsertVulnerability(t *testing.T) {
Name: "TestInsertVulnerability1",
Namespace: n1,
FixedIn: []database.FeatureVersion{f1, f3, f6, f7},
Severity: types.Low,
Severity: database.LowSeverity,
Description: "TestInsertVulnerabilityDescription1",
Link: "TestInsertVulnerabilityLink1",
Metadata: v1meta,
@ -239,7 +232,7 @@ func TestInsertVulnerability(t *testing.T) {
// Update vulnerability.
v1.Description = "TestInsertVulnerabilityLink2"
v1.Link = "TestInsertVulnerabilityLink2"
v1.Severity = types.High
v1.Severity = database.HighSeverity
// Update f3 in f4, add fixed in f5, add fixed in f6 which already exists,
// removes fixed in f7 by adding f8 which is f7 but with MinVersion, and
// add fixed by f5 a second time (duplicated).
@ -288,3 +281,16 @@ func equalsVuln(t *testing.T, expected, actual *database.Vulnerability) {
}
}
}
func TestStringComparison(t *testing.T) {
cmp := compareStringLists([]string{"a", "b", "b", "a"}, []string{"a", "c"})
assert.Len(t, cmp, 1)
assert.NotContains(t, cmp, "a")
assert.Contains(t, cmp, "b")
cmp = compareStringListsInBoth([]string{"a", "a", "b", "c"}, []string{"a", "c", "c"})
assert.Len(t, cmp, 2)
assert.NotContains(t, cmp, "b")
assert.Contains(t, cmp, "a")
assert.Contains(t, cmp, "c")
}

134
database/severity.go Normal file
View File

@ -0,0 +1,134 @@
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package database
import (
"database/sql/driver"
"errors"
"strings"
)
// ErrFailedToParseSeverity is the error returned when a severity could not
// be parsed from a string.
var ErrFailedToParseSeverity = errors.New("failed to parse Severity from input")
// Severity defines a standard scale for measuring the severity of a
// vulnerability.
type Severity string
const (
// UnknownSeverity is either a security problem that has not been assigned to
// a priority yet or a priority that our system did not recognize.
UnknownSeverity Severity = "Unknown"
// NegligibleSeverity is technically a security problem, but is only
// theoretical in nature, requires a very special situation, has almost no
// install base, or does no real damage. These tend not to get backport from
// upstreams, and will likely not be included in security updates unless
// there is an easy fix and some other issue causes an update.
NegligibleSeverity Severity = "Negligible"
// LowSeverity is a security problem, but is hard to exploit due to
// environment, requires a user-assisted attack, a small install base, or
// does very little damage. These tend to be included in security updates
// only when higher priority issues require an update, or if many low
// priority issues have built up.
LowSeverity Severity = "Low"
// MediumSeverity is a real security problem, and is exploitable for many
// people. Includes network daemon denial of service attacks, cross-site
// scripting, and gaining user privileges. Updates should be made soon for
// this priority of issue.
MediumSeverity Severity = "Medium"
// HighSeverity is a real problem, exploitable for many people in a default
// installation. Includes serious remote denial of services, local root
// privilege escalations, or data loss.
HighSeverity Severity = "High"
// CriticalSeverity is a world-burning problem, exploitable for nearly all
// people in a default installation of Linux. Includes remote root privilege
// escalations, or massive data loss.
CriticalSeverity Severity = "Critical"
// Defcon1Severity is a Critical problem which has been manually highlighted
// by the team. It requires an immediate attention.
Defcon1Severity Severity = "Defcon1"
)
// Severities lists all known severities, ordered from lowest to highest.
var Severities = []Severity{
UnknownSeverity,
NegligibleSeverity,
LowSeverity,
MediumSeverity,
HighSeverity,
CriticalSeverity,
Defcon1Severity,
}
// NewSeverity attempts to parse a string into a standard Severity value.
func NewSeverity(s string) (Severity, error) {
for _, ss := range Severities {
if strings.EqualFold(s, string(ss)) {
return ss, nil
}
}
return UnknownSeverity, ErrFailedToParseSeverity
}
// Compare determines the equality of two severities.
//
// If the severities are equal, returns 0.
// If the receiever is less, returns -1.
// If the receiver is greater, returns 1.
func (s Severity) Compare(s2 Severity) int {
var i1, i2 int
for i1 = 0; i1 < len(Severities); i1 = i1 + 1 {
if s == Severities[i1] {
break
}
}
for i2 = 0; i2 < len(Severities); i2 = i2 + 1 {
if s2 == Severities[i2] {
break
}
}
return i1 - i2
}
// Scan implements the database/sql.Scanner interface.
func (s *Severity) Scan(value interface{}) error {
val, ok := value.([]byte)
if !ok {
return errors.New("could not scan a Severity from a non-string input")
}
var err error
*s, err = NewSeverity(string(val))
if err != nil {
return err
}
return nil
}
// Value implements the database/sql/driver.Valuer interface.
func (s Severity) Value() (driver.Value, error) {
return string(s), nil
}

35
database/severity_test.go Normal file
View File

@ -0,0 +1,35 @@
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package database
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestCompareSeverity(t *testing.T) {
assert.Equal(t, MediumSeverity.Compare(MediumSeverity), 0, "Severity comparison failed")
assert.True(t, MediumSeverity.Compare(HighSeverity) < 0, "Severity comparison failed")
assert.True(t, CriticalSeverity.Compare(LowSeverity) > 0, "Severity comparison failed")
}
func TestParseSeverity(t *testing.T) {
_, err := NewSeverity("Test")
assert.Equal(t, ErrFailedToParseSeverity, err)
_, err = NewSeverity("Unknown")
assert.Nil(t, err)
}

18
worker/detectors/feature/apk/apk.go → ext/featurefmt/apk/apk.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,6 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package apk implements a featurefmt.Lister for APK packages.
package apk
import (
@ -21,21 +22,22 @@ import (
"github.com/coreos/pkg/capnslog"
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/featurefmt"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/ext/versionfmt/dpkg"
"github.com/coreos/clair/worker/detectors"
"github.com/coreos/clair/pkg/tarutil"
)
var log = capnslog.NewPackageLogger("github.com/coreos/clair", "worker/detectors/packages")
var log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/featurefmt/apk")
func init() {
detectors.RegisterFeaturesDetector("apk", &detector{})
featurefmt.RegisterLister("apk", &lister{})
}
type detector struct{}
type lister struct{}
func (d *detector) Detect(data map[string][]byte) ([]database.FeatureVersion, error) {
file, exists := data["lib/apk/db/installed"]
func (l lister) ListFeatures(files tarutil.FilesMap) ([]database.FeatureVersion, error) {
file, exists := files["lib/apk/db/installed"]
if !exists {
return []database.FeatureVersion{}, nil
}
@ -83,6 +85,6 @@ func (d *detector) Detect(data map[string][]byte) ([]database.FeatureVersion, er
return pkgs, nil
}
func (d *detector) GetRequiredFiles() []string {
func (l lister) RequiredFilenames() []string {
return []string{"lib/apk/db/installed"}
}

13
worker/detectors/feature/apk/apk_test.go → ext/featurefmt/apk/apk_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -18,11 +18,12 @@ import (
"testing"
"github.com/coreos/clair/database"
"github.com/coreos/clair/worker/detectors/feature"
"github.com/coreos/clair/ext/featurefmt"
"github.com/coreos/clair/pkg/tarutil"
)
func TestAPKFeatureDetection(t *testing.T) {
testData := []feature.TestData{
testData := []featurefmt.TestData{
{
FeatureVersions: []database.FeatureVersion{
{
@ -70,10 +71,10 @@ func TestAPKFeatureDetection(t *testing.T) {
Version: "0.7-r0",
},
},
Data: map[string][]byte{
"lib/apk/db/installed": feature.LoadFileForTest("apk/testdata/installed"),
Files: tarutil.FilesMap{
"lib/apk/db/installed": featurefmt.LoadFileForTest("apk/testdata/installed"),
},
},
}
feature.TestDetector(t, &detector{}, testData)
featurefmt.TestLister(t, &lister{}, testData)
}

0
worker/detectors/feature/apk/testdata/installed → ext/featurefmt/apk/testdata/installed vendored
View File

22
worker/detectors/feature/dpkg/dpkg.go → ext/featurefmt/dpkg/dpkg.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,6 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package dpkg implements a featurefmt.Lister for dpkg packages.
package dpkg
import (
@ -22,28 +23,27 @@ import (
"github.com/coreos/pkg/capnslog"
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/featurefmt"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/ext/versionfmt/dpkg"
"github.com/coreos/clair/worker/detectors"
"github.com/coreos/clair/pkg/tarutil"
)
var (
log = capnslog.NewPackageLogger("github.com/coreos/clair", "worker/detectors/packages")
log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/featurefmt/dpkg")
dpkgSrcCaptureRegexp = regexp.MustCompile(`Source: (?P<name>[^\s]*)( \((?P<version>.*)\))?`)
dpkgSrcCaptureRegexpNames = dpkgSrcCaptureRegexp.SubexpNames()
)
// DpkgFeaturesDetector implements FeaturesDetector and detects dpkg packages
type DpkgFeaturesDetector struct{}
type lister struct{}
func init() {
detectors.RegisterFeaturesDetector("dpkg", &DpkgFeaturesDetector{})
featurefmt.RegisterLister("dpkg", &lister{})
}
// Detect detects packages using var/lib/dpkg/status from the input data
func (detector *DpkgFeaturesDetector) Detect(data map[string][]byte) ([]database.FeatureVersion, error) {
f, hasFile := data["var/lib/dpkg/status"]
func (l lister) ListFeatures(files tarutil.FilesMap) ([]database.FeatureVersion, error) {
f, hasFile := files["var/lib/dpkg/status"]
if !hasFile {
return []database.FeatureVersion{}, nil
}
@ -116,8 +116,6 @@ func (detector *DpkgFeaturesDetector) Detect(data map[string][]byte) ([]database
return packages, nil
}
// GetRequiredFiles returns the list of files required for Detect, without
// leading /
func (detector *DpkgFeaturesDetector) GetRequiredFiles() []string {
func (l lister) RequiredFilenames() []string {
return []string{"var/lib/dpkg/status"}
}

13
worker/detectors/feature/dpkg/dpkg_test.go → ext/featurefmt/dpkg/dpkg_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -18,11 +18,12 @@ import (
"testing"
"github.com/coreos/clair/database"
"github.com/coreos/clair/worker/detectors/feature"
"github.com/coreos/clair/ext/featurefmt"
"github.com/coreos/clair/pkg/tarutil"
)
func TestDpkgFeatureDetection(t *testing.T) {
testData := []feature.TestData{
testData := []featurefmt.TestData{
// Test an Ubuntu dpkg status file
{
FeatureVersions: []database.FeatureVersion{
@ -40,11 +41,11 @@ func TestDpkgFeatureDetection(t *testing.T) {
Version: "5.1.1-12ubuntu1", // The version comes from the "Source:" line
},
},
Data: map[string][]byte{
"var/lib/dpkg/status": feature.LoadFileForTest("dpkg/testdata/status"),
Files: tarutil.FilesMap{
"var/lib/dpkg/status": featurefmt.LoadFileForTest("dpkg/testdata/status"),
},
},
}
feature.TestDetector(t, &DpkgFeaturesDetector{}, testData)
featurefmt.TestLister(t, &lister{}, testData)
}

0
worker/detectors/feature/dpkg/testdata/status → ext/featurefmt/dpkg/testdata/status vendored
View File

126
ext/featurefmt/driver.go Normal file
View File

@ -0,0 +1,126 @@
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package featurefmt exposes functions to dynamically register methods for
// determining the features present in an image layer.
package featurefmt
import (
"io/ioutil"
"path/filepath"
"runtime"
"sync"
"testing"
"github.com/coreos/clair/database"
"github.com/coreos/clair/pkg/tarutil"
"github.com/stretchr/testify/assert"
)
var (
listersM sync.RWMutex
listers = make(map[string]Lister)
)
// Lister represents an ability to list the features present in an image layer.
type Lister interface {
// ListFeatures produces a list of FeatureVersions present in an image layer.
ListFeatures(tarutil.FilesMap) ([]database.FeatureVersion, error)
// RequiredFilenames returns the list of files required to be in the FilesMap
// provided to the ListFeatures method.
//
// Filenames must not begin with "/".
RequiredFilenames() []string
}
// RegisterLister makes a Lister available by the provided name.
//
// If called twice with the same name, the name is blank, or if the provided
// Lister is nil, this function panics.
func RegisterLister(name string, l Lister) {
if name == "" {
panic("featurefmt: could not register a Lister with an empty name")
}
if l == nil {
panic("featurefmt: could not register a nil Lister")
}
listersM.Lock()
defer listersM.Unlock()
if _, dup := listers[name]; dup {
panic("featurefmt: RegisterLister called twice for " + name)
}
listers[name] = l
}
// ListFeatures produces the list of FeatureVersions in an image layer using
// every registered Lister.
func ListFeatures(files tarutil.FilesMap) ([]database.FeatureVersion, error) {
listersM.RLock()
defer listersM.RUnlock()
var totalFeatures []database.FeatureVersion
for _, lister := range listers {
features, err := lister.ListFeatures(files)
if err != nil {
return []database.FeatureVersion{}, err
}
totalFeatures = append(totalFeatures, features...)
}
return totalFeatures, nil
}
// RequiredFilenames returns the total list of files required for all
// registered Listers.
func RequiredFilenames() (files []string) {
listersM.RLock()
defer listersM.RUnlock()
for _, lister := range listers {
files = append(files, lister.RequiredFilenames()...)
}
return
}
// TestData represents the data used to test an implementation of Lister.
type TestData struct {
Files tarutil.FilesMap
FeatureVersions []database.FeatureVersion
}
// LoadFileForTest can be used in order to obtain the []byte contents of a file
// that is meant to be used for test data.
func LoadFileForTest(name string) []byte {
_, filename, _, _ := runtime.Caller(0)
d, _ := ioutil.ReadFile(filepath.Join(filepath.Dir(filename)) + "/" + name)
return d
}
// TestLister runs a Lister on each provided instance of TestData and asserts
// the ouput to be equal to the expected output.
func TestLister(t *testing.T, l Lister, testData []TestData) {
for _, td := range testData {
featureVersions, err := l.ListFeatures(td.Files)
if assert.Nil(t, err) && assert.Len(t, featureVersions, len(td.FeatureVersions)) {
for _, expectedFeatureVersion := range td.FeatureVersions {
assert.Contains(t, featureVersions, expectedFeatureVersion)
}
}
}
}

37
worker/detectors/feature/rpm/rpm.go → ext/featurefmt/rpm/rpm.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,37 +12,36 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package rpm implements a featurefmt.Lister for rpm packages.
package rpm
import (
"bufio"
"io/ioutil"
"os"
"os/exec"
"strings"
"github.com/coreos/pkg/capnslog"
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/featurefmt"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/ext/versionfmt/rpm"
"github.com/coreos/clair/utils"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/worker/detectors"
"github.com/coreos/clair/pkg/commonerr"
"github.com/coreos/clair/pkg/tarutil"
)
var log = capnslog.NewPackageLogger("github.com/coreos/clair", "rpm")
var log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/featurefmt/rpm")
// RpmFeaturesDetector implements FeaturesDetector and detects rpm packages
// It requires the "rpm" binary to be in the PATH
type RpmFeaturesDetector struct{}
type lister struct{}
func init() {
detectors.RegisterFeaturesDetector("rpm", &RpmFeaturesDetector{})
featurefmt.RegisterLister("rpm", &lister{})
}
// Detect detects packages using var/lib/rpm/Packages from the input data
func (detector *RpmFeaturesDetector) Detect(data map[string][]byte) ([]database.FeatureVersion, error) {
f, hasFile := data["var/lib/rpm/Packages"]
func (l lister) ListFeatures(files tarutil.FilesMap) ([]database.FeatureVersion, error) {
f, hasFile := files["var/lib/rpm/Packages"]
if !hasFile {
return []database.FeatureVersion{}, nil
}
@ -55,19 +54,17 @@ func (detector *RpmFeaturesDetector) Detect(data map[string][]byte) ([]database.
defer os.RemoveAll(tmpDir)
if err != nil {
log.Errorf("could not create temporary folder for RPM detection: %s", err)
return []database.FeatureVersion{}, cerrors.ErrFilesystem
return []database.FeatureVersion{}, commonerr.ErrFilesystem
}
err = ioutil.WriteFile(tmpDir+"/Packages", f, 0700)
if err != nil {
log.Errorf("could not create temporary file for RPM detection: %s", err)
return []database.FeatureVersion{}, cerrors.ErrFilesystem
return []database.FeatureVersion{}, commonerr.ErrFilesystem
}
// Query RPM
// We actually extract binary package names instead of source package names here because RHSA refers to package names
// In the dpkg system, we extract the source instead
out, err := utils.Exec(tmpDir, "rpm", "--dbpath", tmpDir, "-qa", "--qf", "%{NAME} %{EPOCH}:%{VERSION}-%{RELEASE}\n")
// Extract binary package names because RHSA refers to binary package names.
out, err := exec.Command("rpm", "--dbpath", tmpDir, "-qa", "--qf", "%{NAME} %{EPOCH}:%{VERSION}-%{RELEASE}\n").CombinedOutput()
if err != nil {
log.Errorf("could not query RPM: %s. output: %s", err, string(out))
// Do not bubble up because we probably won't be able to fix it,
@ -116,8 +113,6 @@ func (detector *RpmFeaturesDetector) Detect(data map[string][]byte) ([]database.
return packages, nil
}
// GetRequiredFiles returns the list of files required for Detect, without
// leading /
func (detector *RpmFeaturesDetector) GetRequiredFiles() []string {
func (l lister) RequiredFilenames() []string {
return []string{"var/lib/rpm/Packages"}
}

13
worker/detectors/feature/rpm/rpm_test.go → ext/featurefmt/rpm/rpm_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -18,11 +18,12 @@ import (
"testing"
"github.com/coreos/clair/database"
"github.com/coreos/clair/worker/detectors/feature"
"github.com/coreos/clair/ext/featurefmt"
"github.com/coreos/clair/pkg/tarutil"
)
func TestRpmFeatureDetection(t *testing.T) {
testData := []feature.TestData{
testData := []featurefmt.TestData{
// Test a CentOS 7 RPM database
// Memo: Use the following command on a RPM-based system to shrink a database: rpm -qa --qf "%{NAME}\n" |tail -n +3| xargs rpm -e --justdb
{
@ -38,11 +39,11 @@ func TestRpmFeatureDetection(t *testing.T) {
Version: "3.2-18.el7",
},
},
Data: map[string][]byte{
"var/lib/rpm/Packages": feature.LoadFileForTest("rpm/testdata/Packages"),
Files: tarutil.FilesMap{
"var/lib/rpm/Packages": featurefmt.LoadFileForTest("rpm/testdata/Packages"),
},
},
}
feature.TestDetector(t, &RpmFeaturesDetector{}, testData)
featurefmt.TestLister(t, &lister{}, testData)
}

0
worker/detectors/feature/rpm/testdata/Packages → ext/featurefmt/rpm/testdata/Packages vendored
View File

21
worker/detectors/namespace/alpinerelease/alpinerelease.go → ext/featurens/alpinerelease/alpinerelease.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,6 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package alpinerelease implements a featurens.Detector for Alpine Linux based
// container image layers.
package alpinerelease
import (
@ -21,8 +23,9 @@ import (
"strings"
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/featurens"
"github.com/coreos/clair/ext/versionfmt/dpkg"
"github.com/coreos/clair/worker/detectors"
"github.com/coreos/clair/pkg/tarutil"
)
const (
@ -33,15 +36,13 @@ const (
var versionRegexp = regexp.MustCompile(`^(\d)+\.(\d)+\.(\d)+$`)
func init() {
detectors.RegisterNamespaceDetector("alpine-release", &detector{})
featurens.RegisterDetector("alpine-release", &detector{})
}
// detector implements NamespaceDetector by reading the current version of
// Alpine Linux from /etc/alpine-release.
type detector struct{}
func (d *detector) Detect(data map[string][]byte) *database.Namespace {
file, exists := data[alpineReleasePath]
func (d detector) Detect(files tarutil.FilesMap) (*database.Namespace, error) {
file, exists := files[alpineReleasePath]
if exists {
scanner := bufio.NewScanner(bytes.NewBuffer(file))
for scanner.Scan() {
@ -52,14 +53,14 @@ func (d *detector) Detect(data map[string][]byte) *database.Namespace {
return &database.Namespace{
Name: osName + ":" + "v" + versionNumbers[0] + "." + versionNumbers[1],
VersionFormat: dpkg.ParserName,
}
}, nil
}
}
}
return nil
return nil, nil
}
func (d *detector) GetRequiredFiles() []string {
func (d detector) RequiredFilenames() []string {
return []string{alpineReleasePath}
}

21
worker/detectors/namespace/alpinerelease/alpinerelease_test.go → ext/featurens/alpinerelease/alpinerelease_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -18,34 +18,35 @@ import (
"testing"
"github.com/coreos/clair/database"
"github.com/coreos/clair/worker/detectors/namespace"
"github.com/coreos/clair/ext/featurens"
"github.com/coreos/clair/pkg/tarutil"
)
func TestAlpineReleaseNamespaceDetection(t *testing.T) {
testData := []namespace.TestData{
func TestDetector(t *testing.T) {
testData := []featurens.TestData{
{
ExpectedNamespace: &database.Namespace{Name: "alpine:v3.3"},
Data: map[string][]byte{"etc/alpine-release": []byte(`3.3.4`)},
Files: tarutil.FilesMap{"etc/alpine-release": []byte(`3.3.4`)},
},
{
ExpectedNamespace: &database.Namespace{Name: "alpine:v3.4"},
Data: map[string][]byte{"etc/alpine-release": []byte(`3.4.0`)},
Files: tarutil.FilesMap{"etc/alpine-release": []byte(`3.4.0`)},
},
{
ExpectedNamespace: &database.Namespace{Name: "alpine:v0.3"},
Data: map[string][]byte{"etc/alpine-release": []byte(`0.3.4`)},
Files: tarutil.FilesMap{"etc/alpine-release": []byte(`0.3.4`)},
},
{
ExpectedNamespace: &database.Namespace{Name: "alpine:v0.3"},
Data: map[string][]byte{"etc/alpine-release": []byte(`
Files: tarutil.FilesMap{"etc/alpine-release": []byte(`
0.3.4
`)},
},
{
ExpectedNamespace: nil,
Data: map[string][]byte{},
Files: tarutil.FilesMap{},
},
}
namespace.TestDetector(t, &detector{}, testData)
featurens.TestDetector(t, &detector{}, testData)
}

31
worker/detectors/namespace/aptsources/aptsources.go → ext/featurens/aptsources/aptsources.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,6 +12,11 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package aptsources implements a featurens.Detector for apt based container
// image layers.
//
// This detector is necessary to determine the precise Debian version when it
// is an unstable version for instance.
package aptsources
import (
@ -19,25 +24,21 @@ import (
"strings"
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/featurens"
"github.com/coreos/clair/ext/versionfmt/dpkg"
"github.com/coreos/clair/worker/detectors"
"github.com/coreos/clair/pkg/tarutil"
)
// AptSourcesNamespaceDetector implements NamespaceDetector and detects the Namespace from the
// /etc/apt/sources.list file.
//
// This detector is necessary to determine precise Debian version when it is
// an unstable version for instance.
type AptSourcesNamespaceDetector struct{}
type detector struct{}
func init() {
detectors.RegisterNamespaceDetector("apt-sources", &AptSourcesNamespaceDetector{})
featurens.RegisterDetector("apt-sources", &detector{})
}
func (detector *AptSourcesNamespaceDetector) Detect(data map[string][]byte) *database.Namespace {
f, hasFile := data["etc/apt/sources.list"]
func (d detector) Detect(files tarutil.FilesMap) (*database.Namespace, error) {
f, hasFile := files["etc/apt/sources.list"]
if !hasFile {
return nil
return nil, nil
}
var OS, version string
@ -79,11 +80,11 @@ func (detector *AptSourcesNamespaceDetector) Detect(data map[string][]byte) *dat
return &database.Namespace{
Name: OS + ":" + version,
VersionFormat: dpkg.ParserName,
}, nil
}
}
return nil
return nil, nil
}
func (detector *AptSourcesNamespaceDetector) GetRequiredFiles() []string {
func (d detector) RequiredFilenames() []string {
return []string{"etc/apt/sources.list"}
}

13
worker/detectors/namespace/aptsources/aptsources_test.go → ext/featurens/aptsources/aptsources_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -18,14 +18,15 @@ import (
"testing"
"github.com/coreos/clair/database"
"github.com/coreos/clair/worker/detectors/namespace"
"github.com/coreos/clair/ext/featurens"
"github.com/coreos/clair/pkg/tarutil"
)
func TestAptSourcesNamespaceDetector(t *testing.T) {
testData := []namespace.TestData{
func TestDetector(t *testing.T) {
testData := []featurens.TestData{
{
ExpectedNamespace: &database.Namespace{Name: "debian:unstable"},
Data: map[string][]byte{
Files: tarutil.FilesMap{
"etc/os-release": []byte(
`PRETTY_NAME="Debian GNU/Linux stretch/sid"
NAME="Debian GNU/Linux"
@ -38,5 +39,5 @@ BUG_REPORT_URL="https://bugs.debian.org/"`),
},
}
namespace.TestDetector(t, &AptSourcesNamespaceDetector{}, testData)
featurens.TestDetector(t, &detector{}, testData)
}

127
ext/featurens/driver.go Normal file
View File

@ -0,0 +1,127 @@
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package featurens exposes functions to dynamically register methods for
// determining a namespace for features present in an image layer.
package featurens
import (
"sync"
"testing"
"github.com/coreos/pkg/capnslog"
"github.com/stretchr/testify/assert"
"github.com/coreos/clair/database"
"github.com/coreos/clair/pkg/tarutil"
)
var (
log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/featurens")
detectorsM sync.RWMutex
detectors = make(map[string]Detector)
)
// Detector represents an ability to detect a namespace used for organizing
// features present in an image layer.
type Detector interface {
// Detect attempts to determine a Namespace from a FilesMap of an image
// layer.
Detect(tarutil.FilesMap) (*database.Namespace, error)
// RequiredFilenames returns the list of files required to be in the FilesMap
// provided to the Detect method.
//
// Filenames must not begin with "/".
RequiredFilenames() []string
}
// RegisterDetector makes a detector available by the provided name.
//
// If called twice with the same name, the name is blank, or if the provided
// Detector is nil, this function panics.
func RegisterDetector(name string, d Detector) {
if name == "" {
panic("namespace: could not register a Detector with an empty name")
}
if d == nil {
panic("namespace: could not register a nil Detector")
}
detectorsM.Lock()
defer detectorsM.Unlock()
if _, dup := detectors[name]; dup {
panic("namespace: RegisterDetector called twice for " + name)
}
detectors[name] = d
}
// Detect iterators through all registered Detectors and returns the first
// non-nil detected namespace.
func Detect(files tarutil.FilesMap) (*database.Namespace, error) {
detectorsM.RLock()
defer detectorsM.RUnlock()
for name, detector := range detectors {
namespace, err := detector.Detect(files)
if err != nil {
log.Warningf("failed while attempting to detect namespace %s: %s", name, err)
return nil, err
}
if namespace != nil {
log.Debugf("detected namespace %s: %#v", name, namespace)
return namespace, nil
}
}
return nil, nil
}
// RequiredFilenames returns the total list of files required for all
// registered Detectors.
func RequiredFilenames() (files []string) {
detectorsM.RLock()
defer detectorsM.RUnlock()
for _, detector := range detectors {
files = append(files, detector.RequiredFilenames()...)
}
return
}
// TestData represents the data used to test an implementation of Detector.
type TestData struct {
Files tarutil.FilesMap
ExpectedNamespace *database.Namespace
}
// TestDetector runs a Detector on each provided instance of TestData and
// asserts the output to be equal to the expected output.
func TestDetector(t *testing.T, d Detector, testData []TestData) {
for _, td := range testData {
namespace, err := d.Detect(td.Files)
assert.Nil(t, err)
if namespace == nil {
assert.Equal(t, td.ExpectedNamespace, namespace)
} else {
assert.Equal(t, td.ExpectedNamespace.Name, namespace.Name)
}
}
}

35
worker/detectors/namespace/lsbrelease/lsbrelease.go → ext/featurens/lsbrelease/lsbrelease.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,6 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package lsbrelease implements a featurens.Detector for container image
// layers containing an lsb-release file.
//
// This detector is necessary for detecting Ubuntu Precise.
package lsbrelease
import (
@ -20,9 +24,10 @@ import (
"strings"
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/featurens"
"github.com/coreos/clair/ext/versionfmt/dpkg"
"github.com/coreos/clair/ext/versionfmt/rpm"
"github.com/coreos/clair/worker/detectors"
"github.com/coreos/clair/pkg/tarutil"
)
var (
@ -30,20 +35,16 @@ var (
lsbReleaseVersionRegexp = regexp.MustCompile(`^DISTRIB_RELEASE=(.*)`)
)
// LsbReleaseNamespaceDetector implements NamespaceDetector and detects the
// Namespace from the /etc/lsb-release file.
//
// This detector is necessary for Ubuntu Precise.
type LsbReleaseNamespaceDetector struct{}
type detector struct{}
func init() {
detectors.RegisterNamespaceDetector("lsb-release", &LsbReleaseNamespaceDetector{})
featurens.RegisterDetector("lsb-release", &detector{})
}
func (detector *LsbReleaseNamespaceDetector) Detect(data map[string][]byte) *database.Namespace {
f, hasFile := data["etc/lsb-release"]
func (d detector) Detect(files tarutil.FilesMap) (*database.Namespace, error) {
f, hasFile := files["etc/lsb-release"]
if !hasFile {
return nil
return nil, nil
}
var OS, version string
@ -79,19 +80,19 @@ func (detector *LsbReleaseNamespaceDetector) Detect(data map[string][]byte) *dat
case "centos", "rhel", "fedora", "amzn", "ol", "oracle":
versionFormat = rpm.ParserName
default:
return nil
return nil, nil
}
if OS != "" && version != "" {
return &database.Namespace{
Name: OS + ":" + version,
VersionFormat: versionFormat,
}
}
return nil
}, nil
}
// GetRequiredFiles returns the list of files that are required for Detect()
func (detector *LsbReleaseNamespaceDetector) GetRequiredFiles() []string {
return nil, nil
}
func (d *detector) RequiredFilenames() []string {
return []string{"etc/lsb-release"}
}

15
worker/detectors/namespace/lsbrelease/lsbrelease_test.go → ext/featurens/lsbrelease/lsbrelease_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -18,14 +18,15 @@ import (
"testing"
"github.com/coreos/clair/database"
"github.com/coreos/clair/worker/detectors/namespace"
"github.com/coreos/clair/ext/featurens"
"github.com/coreos/clair/pkg/tarutil"
)
func TestLsbReleaseNamespaceDetector(t *testing.T) {
testData := []namespace.TestData{
func TestDetector(t *testing.T) {
testData := []featurens.TestData{
{
ExpectedNamespace: &database.Namespace{Name: "ubuntu:12.04"},
Data: map[string][]byte{
Files: tarutil.FilesMap{
"etc/lsb-release": []byte(
`DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
@ -35,7 +36,7 @@ DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS"`),
},
{ // We don't care about the minor version of Debian
ExpectedNamespace: &database.Namespace{Name: "debian:7"},
Data: map[string][]byte{
Files: tarutil.FilesMap{
"etc/lsb-release": []byte(
`DISTRIB_ID=Debian
DISTRIB_RELEASE=7.1
@ -45,5 +46,5 @@ DISTRIB_DESCRIPTION="Debian 7.1"`),
},
}
namespace.TestDetector(t, &LsbReleaseNamespaceDetector{}, testData)
featurens.TestDetector(t, &detector{}, testData)
}

53
worker/detectors/namespace/osrelease/osrelease.go → ext/featurens/osrelease/osrelease.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,6 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package osrelease implements a featurens.Detector for container image
// layers containing an os-release file.
//
// This detector is typically useful for detecting Debian or Ubuntu.
package osrelease
import (
@ -20,40 +24,41 @@ import (
"strings"
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/featurens"
"github.com/coreos/clair/ext/versionfmt/dpkg"
"github.com/coreos/clair/ext/versionfmt/rpm"
"github.com/coreos/clair/worker/detectors"
"github.com/coreos/clair/pkg/tarutil"
)
var (
//log = capnslog.NewPackageLogger("github.com/coreos/clair", "worker/detectors/namespace/osrelease")
osReleaseOSRegexp = regexp.MustCompile(`^ID=(.*)`)
osReleaseVersionRegexp = regexp.MustCompile(`^VERSION_ID=(.*)`)
// blacklistFilenames are files that should exclude this detector.
blacklistFilenames = []string{
"etc/oracle-release",
"etc/redhat-release",
"usr/lib/centos-release",
}
)
// OsReleaseNamespaceDetector implements NamespaceDetector and detects the OS from the
// /etc/os-release and usr/lib/os-release files.
type OsReleaseNamespaceDetector struct{}
type detector struct{}
func init() {
detectors.RegisterNamespaceDetector("os-release", &OsReleaseNamespaceDetector{})
featurens.RegisterDetector("os-release", &detector{})
}
// Detect tries to detect OS/Version using "/etc/os-release" and "/usr/lib/os-release"
// Typically for Debian / Ubuntu
// /etc/debian_version can't be used, it does not make any difference between testing and unstable, it returns stretch/sid
func (detector *OsReleaseNamespaceDetector) Detect(data map[string][]byte) *database.Namespace {
func (d detector) Detect(files tarutil.FilesMap) (*database.Namespace, error) {
var OS, version string
for _, filePath := range detector.getExcludeFiles() {
if _, hasFile := data[filePath]; hasFile {
return nil
for _, filePath := range blacklistFilenames {
if _, hasFile := files[filePath]; hasFile {
return nil, nil
}
}
for _, filePath := range detector.GetRequiredFiles() {
f, hasFile := data[filePath]
for _, filePath := range d.RequiredFilenames() {
f, hasFile := files[filePath]
if !hasFile {
continue
}
@ -82,24 +87,18 @@ func (detector *OsReleaseNamespaceDetector) Detect(data map[string][]byte) *data
case "centos", "rhel", "fedora", "amzn", "ol", "oracle":
versionFormat = rpm.ParserName
default:
return nil
return nil, nil
}
if OS != "" && version != "" {
return &database.Namespace{
Name: OS + ":" + version,
VersionFormat: versionFormat,
}, nil
}
}
return nil
return nil, nil
}
// GetRequiredFiles returns the list of files that are required for Detect()
func (detector *OsReleaseNamespaceDetector) GetRequiredFiles() []string {
func (d detector) RequiredFilenames() []string {
return []string{"etc/os-release", "usr/lib/os-release"}
}
// getExcludeFiles returns the list of files that are ought to exclude this detector from Detect()
func (detector *OsReleaseNamespaceDetector) getExcludeFiles() []string {
return []string{"etc/oracle-release", "etc/redhat-release", "usr/lib/centos-release"}
}

17
worker/detectors/namespace/osrelease/osrelease_test.go → ext/featurens/osrelease/osrelease_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -18,14 +18,15 @@ import (
"testing"
"github.com/coreos/clair/database"
"github.com/coreos/clair/worker/detectors/namespace"
"github.com/coreos/clair/ext/featurens"
"github.com/coreos/clair/pkg/tarutil"
)
func TestOsReleaseNamespaceDetector(t *testing.T) {
testData := []namespace.TestData{
func TestDetector(t *testing.T) {
testData := []featurens.TestData{
{
ExpectedNamespace: &database.Namespace{Name: "debian:8"},
Data: map[string][]byte{
Files: tarutil.FilesMap{
"etc/os-release": []byte(
`PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
@ -39,7 +40,7 @@ BUG_REPORT_URL="https://bugs.debian.org/"`),
},
{
ExpectedNamespace: &database.Namespace{Name: "ubuntu:15.10"},
Data: map[string][]byte{
Files: tarutil.FilesMap{
"etc/os-release": []byte(
`NAME="Ubuntu"
VERSION="15.10 (Wily Werewolf)"
@ -54,7 +55,7 @@ BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"`),
},
{ // Doesn't have quotes around VERSION_ID
ExpectedNamespace: &database.Namespace{Name: "fedora:20"},
Data: map[string][]byte{
Files: tarutil.FilesMap{
"etc/os-release": []byte(
`NAME=Fedora
VERSION="20 (Heisenbug)"
@ -73,5 +74,5 @@ REDHAT_SUPPORT_PRODUCT_VERSION=20`),
},
}
namespace.TestDetector(t, &OsReleaseNamespaceDetector{}, testData)
featurens.TestDetector(t, &detector{}, testData)
}

52
worker/detectors/namespace/redhatrelease/redhatrelease.go → ext/featurens/redhatrelease/redhatrelease.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,6 +12,11 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package redhatrelease implements a featurens.Detector for container image
// layers containing an redhat-release-like files.
//
// This detector is typically useful for detecting CentOS and Red-Hat like
// systems.
package redhatrelease
import (
@ -19,77 +24,64 @@ import (
"strings"
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/featurens"
"github.com/coreos/clair/ext/versionfmt/rpm"
"github.com/coreos/clair/worker/detectors"
"github.com/coreos/pkg/capnslog"
"github.com/coreos/clair/pkg/tarutil"
)
var (
log = capnslog.NewPackageLogger("github.com/coreos/clair", "worker/detectors/namespace/redhatrelease")
oracleReleaseRegexp = regexp.MustCompile(`(?P<os>[^\s]*) (Linux Server release) (?P<version>[\d]+)`)
centosReleaseRegexp = regexp.MustCompile(`(?P<os>[^\s]*) (Linux release|release) (?P<version>[\d]+)`)
redhatReleaseRegexp = regexp.MustCompile(`(?P<os>Red Hat Enterprise Linux) (Client release|Server release|Workstation release) (?P<version>[\d]+)`)
)
// RedhatReleaseNamespaceDetector implements NamespaceDetector and detects the OS from the
// /etc/oracle-release, /etc/centos-release, /etc/redhat-release and /etc/system-release files.
//
// Typically for CentOS and Red-Hat like systems
// eg. CentOS release 5.11 (Final)
// eg. CentOS release 6.6 (Final)
// eg. CentOS Linux release 7.1.1503 (Core)
// eg. Oracle Linux Server release 7.3
// eg. Red Hat Enterprise Linux Server release 7.2 (Maipo)
type RedhatReleaseNamespaceDetector struct{}
type detector struct{}
func init() {
detectors.RegisterNamespaceDetector("redhat-release", &RedhatReleaseNamespaceDetector{})
featurens.RegisterDetector("redhat-release", &detector{})
}
func (detector *RedhatReleaseNamespaceDetector) Detect(data map[string][]byte) *database.Namespace {
for _, filePath := range detector.GetRequiredFiles() {
f, hasFile := data[filePath]
func (d detector) Detect(files tarutil.FilesMap) (*database.Namespace, error) {
for _, filePath := range d.RequiredFilenames() {
f, hasFile := files[filePath]
if !hasFile {
continue
}
var r []string
// try for Oracle Linux
// Attempt to match Oracle Linux.
r = oracleReleaseRegexp.FindStringSubmatch(string(f))
if len(r) == 4 {
return &database.Namespace{
Name: strings.ToLower(r[1]) + ":" + r[3],
VersionFormat: rpm.ParserName,
}
}, nil
}
// try for RHEL
// Attempt to match RHEL.
r = redhatReleaseRegexp.FindStringSubmatch(string(f))
if len(r) == 4 {
// TODO(vbatts) this is a hack until https://github.com/coreos/clair/pull/193
// TODO(vbatts): this is a hack until https://github.com/coreos/clair/pull/193
return &database.Namespace{
Name: "centos" + ":" + r[3],
VersionFormat: rpm.ParserName,
}
}, nil
}
// then try centos first
// Atempt to match CentOS.
r = centosReleaseRegexp.FindStringSubmatch(string(f))
if len(r) == 4 {
return &database.Namespace{
Name: strings.ToLower(r[1]) + ":" + r[3],
VersionFormat: rpm.ParserName,
}, nil
}
}
return nil, nil
}
return nil
}
// GetRequiredFiles returns the list of files that are required for Detect()
func (detector *RedhatReleaseNamespaceDetector) GetRequiredFiles() []string {
func (d detector) RequiredFilenames() []string {
return []string{"etc/oracle-release", "etc/centos-release", "etc/redhat-release", "etc/system-release"}
}

19
worker/detectors/namespace/redhatrelease/redhatrelease_test.go → ext/featurens/redhatrelease/redhatrelease_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -18,36 +18,37 @@ import (
"testing"
"github.com/coreos/clair/database"
"github.com/coreos/clair/worker/detectors/namespace"
"github.com/coreos/clair/ext/featurens"
"github.com/coreos/clair/pkg/tarutil"
)
func TestRedhatReleaseNamespaceDetector(t *testing.T) {
testData := []namespace.TestData{
func TestDetector(t *testing.T) {
testData := []featurens.TestData{
{
ExpectedNamespace: &database.Namespace{Name: "oracle:6"},
Data: map[string][]byte{
Files: tarutil.FilesMap{
"etc/oracle-release": []byte(`Oracle Linux Server release 6.8`),
},
},
{
ExpectedNamespace: &database.Namespace{Name: "oracle:7"},
Data: map[string][]byte{
Files: tarutil.FilesMap{
"etc/oracle-release": []byte(`Oracle Linux Server release 7.2`),
},
},
{
ExpectedNamespace: &database.Namespace{Name: "centos:6"},
Data: map[string][]byte{
Files: tarutil.FilesMap{
"etc/centos-release": []byte(`CentOS release 6.6 (Final)`),
},
},
{
ExpectedNamespace: &database.Namespace{Name: "centos:7"},
Data: map[string][]byte{
Files: tarutil.FilesMap{
"etc/system-release": []byte(`CentOS Linux release 7.1.1503 (Core)`),
},
},
}
namespace.TestDetector(t, &RedhatReleaseNamespaceDetector{}, testData)
featurens.TestDetector(t, &detector{}, testData)
}

42
ext/imagefmt/aci/aci.go Normal file
View File

@ -0,0 +1,42 @@
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package aci implements an imagefmt.Extractor for appc formatted container
// image layers.
package aci
import (
"io"
"path/filepath"
"github.com/coreos/clair/ext/imagefmt"
"github.com/coreos/clair/pkg/tarutil"
)
type format struct{}
func init() {
imagefmt.RegisterExtractor("aci", &format{})
}
func (f format) ExtractFiles(layerReader io.ReadCloser, toExtract []string) (tarutil.FilesMap, error) {
// All contents are inside a "rootfs" directory, so this needs to be
// prepended to each filename.
var filenames []string
for _, filename := range toExtract {
filenames = append(filenames, filepath.Join("rootfs/", filename))
}
return tarutil.ExtractFiles(layerReader, filenames)
}

24
utils/types/priority_test.go → ext/imagefmt/docker/docker.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,21 +12,23 @@
// See the License for the specific language governing permissions and
// limitations under the License.
package types
// Package docker implements an imagefmt.Extractor for docker formatted
// container image layers.
package docker
import (
"testing"
"io"
"github.com/stretchr/testify/assert"
"github.com/coreos/clair/ext/imagefmt"
"github.com/coreos/clair/pkg/tarutil"
)
func TestComparePriority(t *testing.T) {
assert.Equal(t, Medium.Compare(Medium), 0, "Priority comparison failed")
assert.True(t, Medium.Compare(High) < 0, "Priority comparison failed")
assert.True(t, Critical.Compare(Low) > 0, "Priority comparison failed")
type format struct{}
func init() {
imagefmt.RegisterExtractor("docker", &format{})
}
func TestIsValid(t *testing.T) {
assert.False(t, Priority("Test").IsValid())
assert.True(t, Unknown.IsValid())
func (f format) ExtractFiles(layerReader io.ReadCloser, toExtract []string) (tarutil.FilesMap, error) {
return tarutil.ExtractFiles(layerReader, toExtract)
}

150
ext/imagefmt/driver.go Normal file
View File

@ -0,0 +1,150 @@
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package notification fetches notifications from the database and informs the
// specified remote handler about their existences, inviting the third party to
// actively query the API about it.
// Package imagefmt exposes functions to dynamically register methods to
// detect different types of container image formats.
package imagefmt
import (
"fmt"
"io"
"math"
"net/http"
"os"
"strings"
"sync"
"github.com/coreos/clair/pkg/commonerr"
"github.com/coreos/clair/pkg/tarutil"
"github.com/coreos/pkg/capnslog"
)
var (
// ErrCouldNotFindLayer is returned when we could not download or open the layer file.
ErrCouldNotFindLayer = commonerr.NewBadRequestError("could not find layer")
log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/imagefmt")
extractorsM sync.RWMutex
extractors = make(map[string]Extractor)
)
// Extractor represents an ability to extract files from a particular container
// image format.
type Extractor interface {
// ExtractFiles produces a tarutil.FilesMap from a image layer.
ExtractFiles(layer io.ReadCloser, filenames []string) (tarutil.FilesMap, error)
}
// RegisterExtractor makes an extractor available by the provided name.
//
// If called twice with the same name, the name is blank, or if the provided
// Extractor is nil, this function panics.
func RegisterExtractor(name string, d Extractor) {
extractorsM.Lock()
defer extractorsM.Unlock()
if name == "" {
panic("imagefmt: could not register an Extractor with an empty name")
}
if d == nil {
panic("imagefmt: could not register a nil Extractor")
}
// Enforce lowercase names, so that they can be reliably be found in a map.
name = strings.ToLower(name)
if _, dup := extractors[name]; dup {
panic("imagefmt: RegisterExtractor called twice for " + name)
}
extractors[name] = d
}
// Extractors returns the list of the registered extractors.
func Extractors() map[string]Extractor {
extractorsM.RLock()
defer extractorsM.RUnlock()
ret := make(map[string]Extractor)
for k, v := range extractors {
ret[k] = v
}
return ret
}
// UnregisterExtractor removes a Extractor with a particular name from the list.
func UnregisterExtractor(name string) {
extractorsM.Lock()
defer extractorsM.Unlock()
delete(extractors, name)
}
// Extract streams an image layer from disk or over HTTP, determines the
// image format, then extracts the files specified.
func Extract(format, path string, headers map[string]string, toExtract []string) (tarutil.FilesMap, error) {
var layerReader io.ReadCloser
if strings.HasPrefix(path, "http://") || strings.HasPrefix(path, "https://") {
// Create a new HTTP request object.
request, err := http.NewRequest("GET", path, nil)
if err != nil {
return nil, ErrCouldNotFindLayer
}
// Set any provided HTTP Headers.
if headers != nil {
for k, v := range headers {
request.Header.Set(k, v)
}
}
// Send the request and handle the response.
r, err := http.DefaultClient.Do(request)
if err != nil {
log.Warningf("could not download layer: %s", err)
return nil, ErrCouldNotFindLayer
}
// Fail if we don't receive a 2xx HTTP status code.
if math.Floor(float64(r.StatusCode/100)) != 2 {
log.Warningf("could not download layer: got status code %d, expected 2XX", r.StatusCode)
return nil, ErrCouldNotFindLayer
}
layerReader = r.Body
} else {
var err error
layerReader, err = os.Open(path)
if err != nil {
return nil, ErrCouldNotFindLayer
}
}
defer layerReader.Close()
if extractor, exists := Extractors()[strings.ToLower(format)]; exists {
files, err := extractor.ExtractFiles(layerReader, toExtract)
if err != nil {
return nil, err
}
return files, nil
}
return nil, commonerr.NewBadRequestError(fmt.Sprintf("unsupported image format '%s'", format))
}

99
ext/notification/driver.go Normal file
View File

@ -0,0 +1,99 @@
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package notification fetches notifications from the database and informs the
// specified remote handler about their existences, inviting the third party to
// actively query the API about it.
// Package notification exposes functions to dynamically register methods to
// deliver notifications from the Clair database.
package notification
import (
"sync"
"time"
"github.com/coreos/pkg/capnslog"
"github.com/coreos/clair/database"
)
var (
log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/notification")
sendersM sync.RWMutex
senders = make(map[string]Sender)
)
// Config is the configuration for the Notifier service and its registered
// notifiers.
type Config struct {
Attempts int
RenotifyInterval time.Duration
Params map[string]interface{} `yaml:",inline"`
}
// Sender represents anything that can transmit notifications.
type Sender interface {
// Configure attempts to initialize the notifier with the provided configuration.
// It returns whether the notifier is enabled or not.
Configure(*Config) (bool, error)
// Send informs the existence of the specified notification.
Send(notification database.VulnerabilityNotification) error
}
// RegisterSender makes a Sender available by the provided name.
//
// If called twice with the same name, the name is blank, or if the provided
// Sender is nil, this function panics.
func RegisterSender(name string, s Sender) {
if name == "" {
panic("notification: could not register a Sender with an empty name")
}
if s == nil {
panic("notification: could not register a nil Sender")
}
sendersM.Lock()
defer sendersM.Unlock()
if _, dup := senders[name]; dup {
panic("notification: RegisterSender called twice for " + name)
}
senders[name] = s
}
// Senders returns the list of the registered Senders.
func Senders() map[string]Sender {
sendersM.RLock()
defer sendersM.RUnlock()
ret := make(map[string]Sender)
for k, v := range senders {
ret[k] = v
}
return ret
}
// UnregisterSender removes a Sender with a particular name from the list.
func UnregisterSender(name string) {
sendersM.Lock()
defer sendersM.Unlock()
delete(senders, name)
}

34
notifier/notifiers/webhook.go → ext/notification/webhook/webhook.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,8 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package notifiers implements several kinds of notifier.Notifier
package notifiers
// Package webhook implements a notification sender for HTTP JSON webhooks.
package webhook
import (
"bytes"
@ -29,21 +29,19 @@ import (
"gopkg.in/yaml.v2"
"github.com/coreos/clair/config"
"github.com/coreos/clair/database"
"github.com/coreos/clair/notifier"
"github.com/coreos/clair/ext/notification"
)
const timeout = 5 * time.Second
// A WebhookNotifier dispatches notifications to a webhook endpoint.
type WebhookNotifier struct {
type sender struct {
endpoint string
client *http.Client
}
// A WebhookNotifierConfiguration represents the configuration of a WebhookNotifier.
type WebhookNotifierConfiguration struct {
// Config represents the configuration of a Webhook Sender.
type Config struct {
Endpoint string
ServerName string
CertFile string
@ -53,12 +51,12 @@ type WebhookNotifierConfiguration struct {
}
func init() {
notifier.RegisterNotifier("webhook", &WebhookNotifier{})
notification.RegisterSender("webhook", &sender{})
}
func (h *WebhookNotifier) Configure(config *config.NotifierConfig) (bool, error) {
func (s *sender) Configure(config *notification.Config) (bool, error) {
// Get configuration
var httpConfig WebhookNotifierConfiguration
var httpConfig Config
if config == nil {
return false, nil
}
@ -81,11 +79,11 @@ func (h *WebhookNotifier) Configure(config *config.NotifierConfig) (bool, error)
if _, err := url.ParseRequestURI(httpConfig.Endpoint); err != nil {
return false, fmt.Errorf("could not parse endpoint URL: %s\n", err)
}
h.endpoint = httpConfig.Endpoint
s.endpoint = httpConfig.Endpoint
// Setup HTTP client.
transport := &http.Transport{}
h.client = &http.Client{
s.client = &http.Client{
Transport: transport,
Timeout: timeout,
}
@ -114,7 +112,7 @@ type notificationEnvelope struct {
}
}
func (h *WebhookNotifier) Send(notification database.VulnerabilityNotification) error {
func (s *sender) Send(notification database.VulnerabilityNotification) error {
// Marshal notification.
jsonNotification, err := json.Marshal(notificationEnvelope{struct{ Name string }{notification.Name}})
if err != nil {
@ -122,7 +120,7 @@ func (h *WebhookNotifier) Send(notification database.VulnerabilityNotification)
}
// Send notification via HTTP POST.
resp, err := h.client.Post(h.endpoint, "application/json", bytes.NewBuffer(jsonNotification))
resp, err := s.client.Post(s.endpoint, "application/json", bytes.NewBuffer(jsonNotification))
if err != nil || resp == nil || (resp.StatusCode != 200 && resp.StatusCode != 201) {
if resp != nil {
return fmt.Errorf("got status %d, expected 200/201", resp.StatusCode)
@ -134,11 +132,11 @@ func (h *WebhookNotifier) Send(notification database.VulnerabilityNotification)
return nil
}
// loadTLSClientConfig initializes a *tls.Config using the given WebhookNotifierConfiguration.
// loadTLSClientConfig initializes a *tls.Config using the given Config.
//
// If no certificates are given, (nil, nil) is returned.
// The CA certificate is optional and falls back to the system default.
func loadTLSClientConfig(cfg *WebhookNotifierConfiguration) (*tls.Config, error) {
func loadTLSClientConfig(cfg *Config) (*tls.Config, error) {
if cfg.CertFile == "" || cfg.KeyFile == "" {
return nil, nil
}

4
ext/versionfmt/dpkg/parser.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,6 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package dpkg implements a versionfmt.Parser for version numbers used in dpkg
// based software packages.
package dpkg
import (

10
ext/versionfmt/driver.go
View File

@ -63,18 +63,20 @@ type Parser interface {
// if the provided Parser is nil, this function panics.
func RegisterParser(name string, p Parser) {
if name == "" {
panic("Could not register a Parser with an empty name")
panic("versionfmt: could not register a Parser with an empty name")
}
if p == nil {
panic("Could not register a nil Parser")
panic("versionfmt: could not register a nil Parser")
}
parsersM.Lock()
defer parsersM.Unlock()
if _, alreadyExists := parsers[name]; alreadyExists {
panic("Parser '" + name + "' is already registered")
if _, dup := parsers[name]; dup {
panic("versionfmt: RegisterParser called twice for " + name)
}
parsers[name] = p
}

4
ext/versionfmt/rpm/parser.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,6 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package rpm implements a versionfmt.Parser for version numbers used in rpm
// based software packages.
package rpm
import (

88
ext/vulnmdsrc/driver.go Normal file
View File

@ -0,0 +1,88 @@
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package vulnmdsrc exposes functions to dynamically register vulnerability
// metadata sources used to update a Clair database.
package vulnmdsrc
import (
"sync"
"github.com/coreos/clair/database"
)
var (
appendersM sync.RWMutex
appenders = make(map[string]Appender)
)
// AppendFunc is the type of a callback provided to an Appender.
type AppendFunc func(metadataKey string, metadata interface{}, severity database.Severity)
// Appender represents anything that can fetch vulnerability metadata and
// append it to a Vulnerability.
type Appender interface {
// BuildCache loads metadata into memory such that it can be quickly accessed
// for future calls to Append.
BuildCache(database.Datastore) error
// AddMetadata adds metadata to the given database.Vulnerability.
// It is expected that the fetcher uses .Lock.Lock() when manipulating the Metadata map.
// Append
Append(vulnName string, callback AppendFunc) error
// PurgeCache deallocates metadata from memory after all calls to Append are
// finished.
PurgeCache()
// Clean deletes any allocated resources.
// It is invoked when Clair stops.
Clean()
}
// RegisterAppender makes an Appender available by the provided name.
//
// If called twice with the same name, the name is blank, or if the provided
// Appender is nil, this function panics.
func RegisterAppender(name string, a Appender) {
if name == "" {
panic("vulnmdsrc: could not register an Appender with an empty name")
}
if a == nil {
panic("vulnmdsrc: could not register a nil Appender")
}
appendersM.Lock()
defer appendersM.Unlock()
if _, dup := appenders[name]; dup {
panic("vulnmdsrc: RegisterAppender called twice for " + name)
}
appenders[name] = a
}
// Appenders returns the list of the registered Appenders.
func Appenders() map[string]Appender {
appendersM.RLock()
defer appendersM.RUnlock()
ret := make(map[string]Appender)
for k, v := range appenders {
ret[k] = v
}
return ret
}

36
utils/exec.go → ext/vulnmdsrc/nvd/nested_read_closer.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,28 +12,22 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package utils simply defines utility functions and types.
package utils
package nvd
import (
"bytes"
"os/exec"
)
import "io"
// Exec runs the given binary with arguments
func Exec(dir string, bin string, args ...string) ([]byte, error) {
_, err := exec.LookPath(bin)
if err != nil {
return nil, err
// NestedReadCloser wraps an io.Reader and implements io.ReadCloser by closing every embed
// io.ReadCloser.
// It allows chaining io.ReadCloser together and still keep the ability to close them all in a
// simple manner.
type NestedReadCloser struct {
io.Reader
NestedReadClosers []io.ReadCloser
}
cmd := exec.Command(bin, args...)
cmd.Dir = dir
var buf bytes.Buffer
cmd.Stdout = &buf
cmd.Stderr = &buf
err = cmd.Run()
return buf.Bytes(), err
// Close closes the gzip.Reader and the underlying io.ReadCloser.
func (nrc *NestedReadCloser) Close() {
for _, nestedReadCloser := range nrc.NestedReadClosers {
nestedReadCloser.Close()
}
}

124
updater/metadata_fetchers/nvd/nvd.go → ext/vulnmdsrc/nvd/nvd.go
View File

@ -1,3 +1,19 @@
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package nvd implements a vulnerability metadata appender using the NIST NVD
// database.
package nvd
import (
@ -12,32 +28,27 @@ import (
"os"
"strconv"
"strings"
"sync"
"time"
"github.com/coreos/clair/database"
"github.com/coreos/clair/updater"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/utils/types"
"github.com/coreos/pkg/capnslog"
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/vulnmdsrc"
"github.com/coreos/clair/pkg/commonerr"
)
const (
dataFeedURL string = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-%s.xml.gz"
dataFeedMetaURL string = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-%s.meta"
metadataKey string = "NVD"
appenderName string = "NVD"
)
var (
log = capnslog.NewPackageLogger("github.com/coreos/clair", "updater/fetchers/metadata_fetchers")
)
var log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/vulnmdsrc/nvd")
type NVDMetadataFetcher struct {
type appender struct {
localPath string
dataFeedHashes map[string]string
lock sync.Mutex
metadata map[string]NVDMetadata
}
@ -51,46 +62,43 @@ type NVDmetadataCVSSv2 struct {
}
func init() {
updater.RegisterMetadataFetcher("NVD", &NVDMetadataFetcher{})
vulnmdsrc.RegisterAppender(appenderName, &appender{})
}
func (fetcher *NVDMetadataFetcher) Load(datastore database.Datastore) error {
fetcher.lock.Lock()
defer fetcher.lock.Unlock()
func (a *appender) BuildCache(datastore database.Datastore) error {
var err error
fetcher.metadata = make(map[string]NVDMetadata)
a.metadata = make(map[string]NVDMetadata)
// Init if necessary.
if fetcher.localPath == "" {
if a.localPath == "" {
// Create a temporary folder to store the NVD data and create hashes struct.
if fetcher.localPath, err = ioutil.TempDir(os.TempDir(), "nvd-data"); err != nil {
return cerrors.ErrFilesystem
if a.localPath, err = ioutil.TempDir(os.TempDir(), "nvd-data"); err != nil {
return commonerr.ErrFilesystem
}
fetcher.dataFeedHashes = make(map[string]string)
a.dataFeedHashes = make(map[string]string)
}
// Get data feeds.
dataFeedReaders, dataFeedHashes, err := getDataFeeds(fetcher.dataFeedHashes, fetcher.localPath)
dataFeedReaders, dataFeedHashes, err := getDataFeeds(a.dataFeedHashes, a.localPath)
if err != nil {
return err
}
fetcher.dataFeedHashes = dataFeedHashes
a.dataFeedHashes = dataFeedHashes
// Parse data feeds.
for dataFeedName, dataFeedReader := range dataFeedReaders {
var nvd nvd
if err = xml.NewDecoder(dataFeedReader).Decode(&nvd); err != nil {
log.Errorf("could not decode NVD data feed '%s': %s", dataFeedName, err)
return cerrors.ErrCouldNotParse
return commonerr.ErrCouldNotParse
}
// For each entry of this data feed:
for _, nvdEntry := range nvd.Entries {
// Create metadata entry.
if metadata := nvdEntry.Metadata(); metadata != nil {
fetcher.metadata[nvdEntry.Name] = *metadata
a.metadata[nvdEntry.Name] = *metadata
}
}
@ -100,42 +108,20 @@ func (fetcher *NVDMetadataFetcher) Load(datastore database.Datastore) error {
return nil
}
func (fetcher *NVDMetadataFetcher) AddMetadata(vulnerability *updater.VulnerabilityWithLock) error {
fetcher.lock.Lock()
defer fetcher.lock.Unlock()
if nvdMetadata, ok := fetcher.metadata[vulnerability.Name]; ok {
vulnerability.Lock.Lock()
// Create Metadata map if necessary and assign the NVD metadata.
if vulnerability.Metadata == nil {
vulnerability.Metadata = make(map[string]interface{})
}
vulnerability.Metadata[metadataKey] = nvdMetadata
// Set the Severity using the CVSSv2 Score if none is set yet.
if vulnerability.Severity == "" || vulnerability.Severity == types.Unknown {
vulnerability.Severity = scoreToPriority(nvdMetadata.CVSSv2.Score)
}
vulnerability.Lock.Unlock()
func (a *appender) Append(vulnName string, appendFunc vulnmdsrc.AppendFunc) error {
if nvdMetadata, ok := a.metadata[vulnName]; ok {
appendFunc(appenderName, nvdMetadata, SeverityFromCVSS(nvdMetadata.CVSSv2.Score))
}
return nil
}
func (fetcher *NVDMetadataFetcher) Unload() {
fetcher.lock.Lock()
defer fetcher.lock.Unlock()
fetcher.metadata = nil
func (a *appender) PurgeCache() {
a.metadata = nil
}
func (fetcher *NVDMetadataFetcher) Clean() {
fetcher.lock.Lock()
defer fetcher.lock.Unlock()
os.RemoveAll(fetcher.localPath)
func (a *appender) Clean() {
os.RemoveAll(a.localPath)
}
func getDataFeeds(dataFeedHashes map[string]string, localPath string) (map[string]NestedReadCloser, map[string]string, error) {
@ -178,14 +164,14 @@ func getDataFeeds(dataFeedHashes map[string]string, localPath string) (map[strin
r, err := http.Get(fmt.Sprintf(dataFeedURL, dataFeedName))
if err != nil {
log.Errorf("could not download NVD data feed file '%s': %s", dataFeedName, err)
return dataFeedReaders, dataFeedHashes, cerrors.ErrCouldNotDownload
return dataFeedReaders, dataFeedHashes, commonerr.ErrCouldNotDownload
}
// Un-gzip it.
gr, err := gzip.NewReader(r.Body)
if err != nil {
log.Errorf("could not read NVD data feed file '%s': %s", dataFeedName, err)
return dataFeedReaders, dataFeedHashes, cerrors.ErrCouldNotDownload
return dataFeedReaders, dataFeedHashes, commonerr.ErrCouldNotDownload
}
// Store it to a file at the same time if possible.
@ -231,23 +217,25 @@ func getHashFromMetaURL(metaURL string) (string, error) {
return "", errors.New("invalid .meta file format")
}
// scoreToPriority converts the CVSS Score (0.0 - 10.0) into user-friendy
// types.Priority following the qualitative rating scale available in the
// SeverityFromCVSS converts the CVSS Score (0.0 - 10.0) into a
// database.Severity following the qualitative rating scale available in the
// CVSS v3.0 specification (https://www.first.org/cvss/specification-document),
// Table 14. The Negligible level is set for CVSS scores between [0, 1),
// replacing the specified None level, originally used for a score of 0.
func scoreToPriority(score float64) types.Priority {
// Table 14.
//
// The Negligible level is set for CVSS scores between [0, 1), replacing the
// specified None level, originally used for a score of 0.
func SeverityFromCVSS(score float64) database.Severity {
switch {
case score < 1.0:
return types.Negligible
return database.NegligibleSeverity
case score < 3.9:
return types.Low
return database.LowSeverity
case score < 6.9:
return types.Medium
return database.MediumSeverity
case score < 8.9:
return types.High
return database.HighSeverity
case score <= 10:
return types.Critical
return database.CriticalSeverity
}
return types.Unknown
return database.UnknownSeverity
}

14
updater/metadata_fetchers/nvd/xml.go → ext/vulnmdsrc/nvd/xml.go
View File

@ -1,3 +1,17 @@
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package nvd
import (

83
updater/fetchers/alpine/alpine.go → ext/vulnsrc/alpine/alpine.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,16 +12,16 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package alpine implements a vulnerability Fetcher using the alpine-secdb
// git repository.
// Package alpine implements a vulnerability source updater using the
// alpine-secdb git repository.
package alpine
import (
"errors"
"fmt"
"io"
"io/ioutil"
"os"
"os/exec"
"strings"
"gopkg.in/yaml.v2"
@ -31,10 +31,8 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/ext/versionfmt/dpkg"
"github.com/coreos/clair/updater"
"github.com/coreos/clair/utils"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/utils/types"
"github.com/coreos/clair/ext/vulnsrc"
"github.com/coreos/clair/pkg/commonerr"
)
const (
@ -44,30 +42,22 @@ const (
nvdURLPrefix = "https://cve.mitre.org/cgi-bin/cvename.cgi?name="
)
var (
// ErrFilesystem is returned when a fetcher fails to interact with the local filesystem.
ErrFilesystem = errors.New("updater/fetchers: something went wrong when interacting with the fs")
// ErrGitFailure is returned when a fetcher fails to interact with git.
ErrGitFailure = errors.New("updater/fetchers: something went wrong when interacting with git")
log = capnslog.NewPackageLogger("github.com/coreos/clair", "updater/fetchers/alpine")
)
var log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/vulnsrc/alpine")
func init() {
updater.RegisterFetcher("alpine", &fetcher{})
vulnsrc.RegisterUpdater("alpine", &updater{})
}
type fetcher struct {
type updater struct {
repositoryLocalPath string
}
func (f *fetcher) FetchUpdate(db database.Datastore) (resp updater.FetcherResponse, err error) {
func (u *updater) Update(db database.Datastore) (resp vulnsrc.UpdateResponse, err error) {
log.Info("fetching Alpine vulnerabilities")
// Pull the master branch.
var commit string
commit, err = f.pullRepository()
commit, err = u.pullRepository()
if err != nil {
return
}
@ -90,12 +80,12 @@ func (f *fetcher) FetchUpdate(db database.Datastore) (resp updater.FetcherRespon
}
var namespaces []string
namespaces, err = detectNamespaces(f.repositoryLocalPath)
namespaces, err = detectNamespaces(u.repositoryLocalPath)
// Append any changed vulnerabilities to the response.
for _, namespace := range namespaces {
var vulns []database.Vulnerability
var note string
vulns, note, err = parseVulnsFromNamespace(f.repositoryLocalPath, namespace)
vulns, note, err = parseVulnsFromNamespace(u.repositoryLocalPath, namespace)
if err != nil {
return
}
@ -108,6 +98,12 @@ func (f *fetcher) FetchUpdate(db database.Datastore) (resp updater.FetcherRespon
return
}
func (u *updater) Clean() {
if u.repositoryLocalPath != "" {
os.RemoveAll(u.repositoryLocalPath)
}
}
func detectNamespaces(path string) ([]string, error) {
// Open the root directory.
dir, err := os.Open(path)
@ -163,41 +159,40 @@ func parseVulnsFromNamespace(repositoryPath, namespace string) (vulns []database
return
}
func (f *fetcher) pullRepository() (commit string, err error) {
func (u *updater) pullRepository() (commit string, err error) {
// If the repository doesn't exist, clone it.
if _, pathExists := os.Stat(f.repositoryLocalPath); f.repositoryLocalPath == "" || os.IsNotExist(pathExists) {
if f.repositoryLocalPath, err = ioutil.TempDir(os.TempDir(), "alpine-secdb"); err != nil {
return "", ErrFilesystem
if _, pathExists := os.Stat(u.repositoryLocalPath); u.repositoryLocalPath == "" || os.IsNotExist(pathExists) {
if u.repositoryLocalPath, err = ioutil.TempDir(os.TempDir(), "alpine-secdb"); err != nil {
return "", vulnsrc.ErrFilesystem
}
if out, err := utils.Exec(f.repositoryLocalPath, "git", "clone", secdbGitURL, "."); err != nil {
f.Clean()
cmd := exec.Command("git", "clone", secdbGitURL, ".")
cmd.Dir = u.repositoryLocalPath
if out, err := cmd.CombinedOutput(); err != nil {
u.Clean()
log.Errorf("could not pull alpine-secdb repository: %s. output: %s", err, out)
return "", cerrors.ErrCouldNotDownload
return "", commonerr.ErrCouldNotDownload
}
} else {
// The repository exists and it needs to be refreshed via a pull.
_, err := utils.Exec(f.repositoryLocalPath, "git", "pull")
if err != nil {
return "", ErrGitFailure
// The repository already exists and it needs to be refreshed via a pull.
cmd := exec.Command("git", "pull")
cmd.Dir = u.repositoryLocalPath
if _, err := cmd.CombinedOutput(); err != nil {
return "", vulnsrc.ErrGitFailure
}
}
out, err := utils.Exec(f.repositoryLocalPath, "git", "rev-parse", "HEAD")
cmd := exec.Command("git", "rev-parse", "HEAD")
cmd.Dir = u.repositoryLocalPath
out, err := cmd.CombinedOutput()
if err != nil {
return "", ErrGitFailure
return "", vulnsrc.ErrGitFailure
}
commit = strings.TrimSpace(string(out))
return
}
func (f *fetcher) Clean() {
if f.repositoryLocalPath != "" {
os.RemoveAll(f.repositoryLocalPath)
}
}
type secdb33File struct {
Distro string `yaml:"distroversion"`
Packages []struct {
@ -232,7 +227,7 @@ func parse33YAML(r io.Reader) (vulns []database.Vulnerability, err error) {
vulns = append(vulns, database.Vulnerability{
Name: fix,
Severity: types.Unknown,
Severity: database.UnknownSeverity,
Link: nvdURLPrefix + fix,
FixedIn: []database.FeatureVersion{
{
@ -286,7 +281,7 @@ func parse34YAML(r io.Reader) (vulns []database.Vulnerability, err error) {
for _, vulnStr := range vulnStrs {
var vuln database.Vulnerability
vuln.Severity = types.Unknown
vuln.Severity = database.UnknownSeverity
vuln.Name = vulnStr
vuln.Link = nvdURLPrefix + vulnStr
vuln.FixedIn = []database.FeatureVersion{

0
updater/fetchers/alpine/alpine_test.go → ext/vulnsrc/alpine/alpine_test.go
View File

0
updater/fetchers/alpine/testdata/v33_main.yaml → ext/vulnsrc/alpine/testdata/v33_main.yaml vendored
View File

0
updater/fetchers/alpine/testdata/v34_main.yaml → ext/vulnsrc/alpine/testdata/v34_main.yaml vendored
View File

57
updater/fetchers/debian/debian.go → ext/vulnsrc/debian/debian.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,6 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package debian implements a vulnerability source updater using the Debian
// Security Tracker.
package debian
import (
@ -28,9 +30,8 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/ext/versionfmt/dpkg"
"github.com/coreos/clair/updater"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/utils/types"
"github.com/coreos/clair/ext/vulnsrc"
"github.com/coreos/clair/pkg/commonerr"
)
const (
@ -39,7 +40,7 @@ const (
updaterFlag = "debianUpdater"
)
var log = capnslog.NewPackageLogger("github.com/coreos/clair", "updater/fetchers/debian")
var log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/vulnsrc/debian")
type jsonData map[string]map[string]jsonVuln
@ -54,23 +55,20 @@ type jsonRel struct {
Urgency string `json:"urgency"`
}
// DebianFetcher implements updater.Fetcher for the Debian Security Tracker
// (https://security-tracker.debian.org).
type DebianFetcher struct{}
type updater struct{}
func init() {
updater.RegisterFetcher("debian", &DebianFetcher{})
vulnsrc.RegisterUpdater("debian", &updater{})
}
// FetchUpdate fetches vulnerability updates from the Debian Security Tracker.
func (fetcher *DebianFetcher) FetchUpdate(datastore database.Datastore) (resp updater.FetcherResponse, err error) {
func (u *updater) Update(datastore database.Datastore) (resp vulnsrc.UpdateResponse, err error) {
log.Info("fetching Debian vulnerabilities")
// Download JSON.
r, err := http.Get(url)
if err != nil {
log.Errorf("could not download Debian's update: %s", err)
return resp, cerrors.ErrCouldNotDownload
return resp, commonerr.ErrCouldNotDownload
}
// Get the SHA-1 of the latest update's JSON data
@ -88,7 +86,9 @@ func (fetcher *DebianFetcher) FetchUpdate(datastore database.Datastore) (resp up
return resp, nil
}
func buildResponse(jsonReader io.Reader, latestKnownHash string) (resp updater.FetcherResponse, err error) {
func (u *updater) Clean() {}
func buildResponse(jsonReader io.Reader, latestKnownHash string) (resp vulnsrc.UpdateResponse, err error) {
hash := latestKnownHash
// Defer the addition of flag information to the response.
@ -109,7 +109,7 @@ func buildResponse(jsonReader io.Reader, latestKnownHash string) (resp updater.F
err = json.NewDecoder(teedJSONReader).Decode(&data)
if err != nil {
log.Errorf("could not unmarshal Debian's JSON: %s", err)
return resp, cerrors.ErrCouldNotParse
return resp, commonerr.ErrCouldNotParse
}
// Calculate the hash and skip updating if the hash has been seen before.
@ -157,17 +157,17 @@ func parseDebianJSON(data *jsonData) (vulnerabilities []database.Vulnerability,
vulnerability = &database.Vulnerability{
Name: vulnName,
Link: strings.Join([]string{cveURLPrefix, "/", vulnName}, ""),
Severity: types.Unknown,
Severity: database.UnknownSeverity,
Description: vulnNode.Description,
}
}
// Set the priority of the vulnerability.
// In the JSON, a vulnerability has one urgency per package it affects.
severity := SeverityFromUrgency(releaseNode.Urgency)
if severity.Compare(vulnerability.Severity) > 0 {
// The highest urgency should be the one set.
urgency := urgencyToSeverity(releaseNode.Urgency)
if urgency.Compare(vulnerability.Severity) > 0 {
vulnerability.Severity = urgency
vulnerability.Severity = severity
}
// Determine the version of the package the vulnerability affects.
@ -218,42 +218,41 @@ func parseDebianJSON(data *jsonData) (vulnerabilities []database.Vulnerability,
return
}
func urgencyToSeverity(urgency string) types.Priority {
// SeverityFromUrgency converts the urgency scale used by the Debian Security
// Bug Tracker into a database.Severity.
func SeverityFromUrgency(urgency string) database.Severity {
switch urgency {
case "not yet assigned":
return types.Unknown
return database.UnknownSeverity
case "end-of-life":
fallthrough
case "unimportant":
return types.Negligible
return database.NegligibleSeverity
case "low":
fallthrough
case "low*":
fallthrough
case "low**":
return types.Low
return database.LowSeverity
case "medium":
fallthrough
case "medium*":
fallthrough
case "medium**":
return types.Medium
return database.MediumSeverity
case "high":
fallthrough
case "high*":
fallthrough
case "high**":
return types.High
return database.HighSeverity
default:
log.Warningf("could not determine vulnerability priority from: %s", urgency)
return types.Unknown
log.Warningf("could not determine vulnerability severity from: %s", urgency)
return database.UnknownSeverity
}
}
// Clean deletes any allocated resources.
func (fetcher *DebianFetcher) Clean() {}

9
updater/fetchers/debian/debian_test.go → ext/vulnsrc/debian/debian_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -23,7 +23,6 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/ext/versionfmt/dpkg"
"github.com/coreos/clair/utils/types"
"github.com/stretchr/testify/assert"
)
@ -37,7 +36,7 @@ func TestDebianParser(t *testing.T) {
for _, vulnerability := range response.Vulnerabilities {
if vulnerability.Name == "CVE-2015-1323" {
assert.Equal(t, "https://security-tracker.debian.org/tracker/CVE-2015-1323", vulnerability.Link)
assert.Equal(t, types.Low, vulnerability.Severity)
assert.Equal(t, database.LowSeverity, vulnerability.Severity)
assert.Equal(t, "This vulnerability is not very dangerous.", vulnerability.Description)
expectedFeatureVersions := []database.FeatureVersion{
@ -68,7 +67,7 @@ func TestDebianParser(t *testing.T) {
}
} else if vulnerability.Name == "CVE-2003-0779" {
assert.Equal(t, "https://security-tracker.debian.org/tracker/CVE-2003-0779", vulnerability.Link)
assert.Equal(t, types.High, vulnerability.Severity)
assert.Equal(t, database.HighSeverity, vulnerability.Severity)
assert.Equal(t, "But this one is very dangerous.", vulnerability.Description)
expectedFeatureVersions := []database.FeatureVersion{
@ -109,7 +108,7 @@ func TestDebianParser(t *testing.T) {
}
} else if vulnerability.Name == "CVE-2013-2685" {
assert.Equal(t, "https://security-tracker.debian.org/tracker/CVE-2013-2685", vulnerability.Link)
assert.Equal(t, types.Negligible, vulnerability.Severity)
assert.Equal(t, database.NegligibleSeverity, vulnerability.Severity)
assert.Equal(t, "Un-affected packages.", vulnerability.Description)
expectedFeatureVersions := []database.FeatureVersion{

0
updater/fetchers/debian/testdata/fetcher_debian_test.json → ext/vulnsrc/debian/testdata/fetcher_debian_test.json vendored
View File

90
ext/vulnsrc/driver.go Normal file
View File

@ -0,0 +1,90 @@
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package vulnsrc exposes functions to dynamically register vulnerability
// sources used to update a Clair database.
package vulnsrc
import (
"errors"
"sync"
"github.com/coreos/clair/database"
)
var (
// ErrFilesystem is returned when a fetcher fails to interact with the local filesystem.
ErrFilesystem = errors.New("vulnsrc: something went wrong when interacting with the fs")
// ErrGitFailure is returned when a fetcher fails to interact with git.
ErrGitFailure = errors.New("vulnsrc: something went wrong when interacting with git")
updatersM sync.RWMutex
updaters = make(map[string]Updater)
)
// UpdateResponse represents the sum of results of an update.
type UpdateResponse struct {
FlagName string
FlagValue string
Notes []string
Vulnerabilities []database.Vulnerability
}
// Updater represents anything that can fetch vulnerabilities and insert them
// into a Clair datastore.
type Updater interface {
// Update gets vulnerability updates.
Update(database.Datastore) (UpdateResponse, error)
// Clean deletes any allocated resources.
// It is invoked when Clair stops.
Clean()
}
// RegisterUpdater makes an Updater available by the provided name.
//
// If called twice with the same name, the name is blank, or if the provided
// Updater is nil, this function panics.
func RegisterUpdater(name string, u Updater) {
if name == "" {
panic("vulnsrc: could not register an Updater with an empty name")
}
if u == nil {
panic("vulnsrc: could not register a nil Updater")
}
updatersM.Lock()
defer updatersM.Unlock()
if _, dup := updaters[name]; dup {
panic("vulnsrc: RegisterUpdater called twice for " + name)
}
updaters[name] = u
}
// Updaters returns the list of the registered Updaters.
func Updaters() map[string]Updater {
updatersM.RLock()
defer updatersM.RUnlock()
ret := make(map[string]Updater)
for k, v := range updaters {
ret[k] = v
}
return ret
}

58
updater/fetchers/oracle/oracle.go → ext/vulnsrc/oracle/oracle.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,6 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package oracle implements a vulnerability source updater using the
// Oracle Linux OVAL Database.
package oracle
import (
@ -23,13 +25,13 @@ import (
"strconv"
"strings"
"github.com/coreos/pkg/capnslog"
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/ext/versionfmt/rpm"
"github.com/coreos/clair/updater"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/utils/types"
"github.com/coreos/pkg/capnslog"
"github.com/coreos/clair/ext/vulnsrc"
"github.com/coreos/clair/pkg/commonerr"
)
const (
@ -47,7 +49,7 @@ var (
elsaRegexp = regexp.MustCompile(`com.oracle.elsa-(\d+).xml`)
log = capnslog.NewPackageLogger("github.com/coreos/clair", "updater/fetchers/oracle")
log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/vulnsrc/oracle")
)
type oval struct {
@ -77,16 +79,13 @@ type criterion struct {
Comment string `xml:"comment,attr"`
}
// OracleFetcher implements updater.Fetcher and gets vulnerability updates from
// the Oracle Linux OVAL definitions.
type OracleFetcher struct{}
type updater struct{}
func init() {
updater.RegisterFetcher("Oracle", &OracleFetcher{})
vulnsrc.RegisterUpdater("oracle", &updater{})
}
// FetchUpdate gets vulnerability updates from the Oracle Linux OVAL definitions.
func (f *OracleFetcher) FetchUpdate(datastore database.Datastore) (resp updater.FetcherResponse, err error) {
func (u *updater) Update(datastore database.Datastore) (resp vulnsrc.UpdateResponse, err error) {
log.Info("fetching Oracle Linux vulnerabilities")
// Get the first ELSA we have to manage.
@ -104,7 +103,7 @@ func (f *OracleFetcher) FetchUpdate(datastore database.Datastore) (resp updater.
r, err := http.Get(ovalURI)
if err != nil {
log.Errorf("could not download Oracle's update list: %s", err)
return resp, cerrors.ErrCouldNotDownload
return resp, commonerr.ErrCouldNotDownload
}
defer r.Body.Close()
@ -127,7 +126,7 @@ func (f *OracleFetcher) FetchUpdate(datastore database.Datastore) (resp updater.
r, err := http.Get(ovalURI + elsaFilePrefix + strconv.Itoa(elsa) + ".xml")
if err != nil {
log.Errorf("could not download Oracle's update file: %s", err)
return resp, cerrors.ErrCouldNotDownload
return resp, commonerr.ErrCouldNotDownload
}
// Parse the XML.
@ -153,13 +152,15 @@ func (f *OracleFetcher) FetchUpdate(datastore database.Datastore) (resp updater.
return resp, nil
}
func (u *updater) Clean() {}
func parseELSA(ovalReader io.Reader) (vulnerabilities []database.Vulnerability, err error) {
// Decode the XML.
var ov oval
err = xml.NewDecoder(ovalReader).Decode(&ov)
if err != nil {
log.Errorf("could not decode Oracle's XML: %s", err)
err = cerrors.ErrCouldNotParse
err = commonerr.ErrCouldNotParse
return
}
@ -171,7 +172,7 @@ func parseELSA(ovalReader io.Reader) (vulnerabilities []database.Vulnerability,
vulnerability := database.Vulnerability{
Name: name(definition),
Link: link(definition),
Severity: priority(definition),
Severity: severity(definition),
Description: description(definition),
}
for _, p := range pkgs {
@ -335,27 +336,20 @@ func link(def definition) (link string) {
return
}
func priority(def definition) types.Priority {
// Parse the priority.
priority := strings.ToLower(def.Severity)
// Normalize the priority.
switch priority {
func severity(def definition) database.Severity {
switch strings.ToLower(def.Severity) {
case "n/a":
return types.Negligible
return database.NegligibleSeverity
case "low":
return types.Low
return database.LowSeverity
case "moderate":
return types.Medium
return database.MediumSeverity
case "important":
return types.High
return database.HighSeverity
case "critical":
return types.Critical
return database.CriticalSeverity
default:
log.Warningf("could not determine vulnerability priority from: %s.", priority)
return types.Unknown
log.Warningf("could not determine vulnerability severity from: %s.", def.Severity)
return database.UnknownSeverity
}
}
// Clean deletes any allocated resources.
func (f *OracleFetcher) Clean() {}

7
updater/fetchers/oracle/oracle_test.go → ext/vulnsrc/oracle/oracle_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -22,7 +22,6 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt/rpm"
"github.com/coreos/clair/utils/types"
"github.com/stretchr/testify/assert"
)
@ -38,7 +37,7 @@ func TestOracleParser(t *testing.T) {
if assert.Nil(t, err) && assert.Len(t, vulnerabilities, 1) {
assert.Equal(t, "ELSA-2015-1193", vulnerabilities[0].Name)
assert.Equal(t, "http://linux.oracle.com/errata/ELSA-2015-1193.html", vulnerabilities[0].Link)
assert.Equal(t, types.Medium, vulnerabilities[0].Severity)
assert.Equal(t, database.MediumSeverity, vulnerabilities[0].Severity)
assert.Equal(t, ` [3.1.1-7] Resolves: rhbz#1217104 CVE-2015-0252 `, vulnerabilities[0].Description)
expectedFeatureVersions := []database.FeatureVersion{
@ -86,7 +85,7 @@ func TestOracleParser(t *testing.T) {
if assert.Nil(t, err) && assert.Len(t, vulnerabilities, 1) {
assert.Equal(t, "ELSA-2015-1207", vulnerabilities[0].Name)
assert.Equal(t, "http://linux.oracle.com/errata/ELSA-2015-1207.html", vulnerabilities[0].Link)
assert.Equal(t, types.Critical, vulnerabilities[0].Severity)
assert.Equal(t, database.CriticalSeverity, vulnerabilities[0].Severity)
assert.Equal(t, ` [38.1.0-1.0.1.el7_1] - Add firefox-oracle-default-prefs.js and remove the corresponding Red Hat file [38.1.0-1] - Update to 38.1.0 ESR [38.0.1-2] - Fixed rhbz#1222807 by removing preun section `, vulnerabilities[0].Description)
expectedFeatureVersions := []database.FeatureVersion{
{

0
updater/fetchers/oracle/testdata/fetcher_oracle_test.1.xml → ext/vulnsrc/oracle/testdata/fetcher_oracle_test.1.xml vendored
View File

0
updater/fetchers/oracle/testdata/fetcher_oracle_test.2.xml → ext/vulnsrc/oracle/testdata/fetcher_oracle_test.2.xml vendored
View File

53
updater/fetchers/rhel/rhel.go → ext/vulnsrc/rhel/rhel.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,6 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package rhel implements a vulnerability source updater using the
// Red Hat Linux OVAL Database.
package rhel
import (
@ -28,9 +30,8 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/ext/versionfmt/rpm"
"github.com/coreos/clair/updater"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/utils/types"
"github.com/coreos/clair/ext/vulnsrc"
"github.com/coreos/clair/pkg/commonerr"
)
const (
@ -82,17 +83,14 @@ type criterion struct {
Comment string `xml:"comment,attr"`
}
// RHELFetcher implements updater.Fetcher and gets vulnerability updates from
// the Red Hat OVAL definitions.
type RHELFetcher struct{}
type updater struct{}
func init() {
updater.RegisterFetcher("Red Hat", &RHELFetcher{})
vulnsrc.RegisterUpdater("rhel", &updater{})
}
// FetchUpdate gets vulnerability updates from the Red Hat OVAL definitions.
func (f *RHELFetcher) FetchUpdate(datastore database.Datastore) (resp updater.FetcherResponse, err error) {
log.Info("fetching Red Hat vulnerabilities")
func (u *updater) Update(datastore database.Datastore) (resp vulnsrc.UpdateResponse, err error) {
log.Info("fetching RHEL vulnerabilities")
// Get the first RHSA we have to manage.
flagValue, err := datastore.GetKeyValue(updaterFlag)
@ -108,7 +106,7 @@ func (f *RHELFetcher) FetchUpdate(datastore database.Datastore) (resp updater.Fe
r, err := http.Get(ovalURI)
if err != nil {
log.Errorf("could not download RHEL's update list: %s", err)
return resp, cerrors.ErrCouldNotDownload
return resp, commonerr.ErrCouldNotDownload
}
// Get the list of RHSAs that we have to process.
@ -130,7 +128,7 @@ func (f *RHELFetcher) FetchUpdate(datastore database.Datastore) (resp updater.Fe
r, err := http.Get(ovalURI + rhsaFilePrefix + strconv.Itoa(rhsa) + ".xml")
if err != nil {
log.Errorf("could not download RHEL's update file: %s", err)
return resp, cerrors.ErrCouldNotDownload
return resp, commonerr.ErrCouldNotDownload
}
// Parse the XML.
@ -156,13 +154,15 @@ func (f *RHELFetcher) FetchUpdate(datastore database.Datastore) (resp updater.Fe
return resp, nil
}
func (u *updater) Clean() {}
func parseRHSA(ovalReader io.Reader) (vulnerabilities []database.Vulnerability, err error) {
// Decode the XML.
var ov oval
err = xml.NewDecoder(ovalReader).Decode(&ov)
if err != nil {
log.Errorf("could not decode RHEL's XML: %s", err)
err = cerrors.ErrCouldNotParse
err = commonerr.ErrCouldNotParse
return
}
@ -174,7 +174,7 @@ func parseRHSA(ovalReader io.Reader) (vulnerabilities []database.Vulnerability,
vulnerability := database.Vulnerability{
Name: name(definition),
Link: link(definition),
Severity: priority(definition),
Severity: severity(definition),
Description: description(definition),
}
for _, p := range pkgs {
@ -343,25 +343,18 @@ func link(def definition) (link string) {
return
}
func priority(def definition) types.Priority {
// Parse the priority.
priority := strings.TrimSpace(def.Title[strings.LastIndex(def.Title, "(")+1 : len(def.Title)-1])
// Normalize the priority.
switch priority {
func severity(def definition) database.Severity {
switch strings.TrimSpace(def.Title[strings.LastIndex(def.Title, "(")+1 : len(def.Title)-1]) {
case "Low":
return types.Low
return database.LowSeverity
case "Moderate":
return types.Medium
return database.MediumSeverity
case "Important":
return types.High
return database.HighSeverity
case "Critical":
return types.Critical
return database.CriticalSeverity
default:
log.Warning("could not determine vulnerability priority from: %s.", priority)
return types.Unknown
log.Warning("could not determine vulnerability severity from: %s.", def.Title)
return database.UnknownSeverity
}
}
// Clean deletes any allocated resources.
func (f *RHELFetcher) Clean() {}

7
updater/fetchers/rhel/rhel_test.go → ext/vulnsrc/rhel/rhel_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -22,7 +22,6 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt/rpm"
"github.com/coreos/clair/utils/types"
"github.com/stretchr/testify/assert"
)
@ -36,7 +35,7 @@ func TestRHELParser(t *testing.T) {
if assert.Nil(t, err) && assert.Len(t, vulnerabilities, 1) {
assert.Equal(t, "RHSA-2015:1193", vulnerabilities[0].Name)
assert.Equal(t, "https://rhn.redhat.com/errata/RHSA-2015-1193.html", vulnerabilities[0].Link)
assert.Equal(t, types.Medium, vulnerabilities[0].Severity)
assert.Equal(t, database.MediumSeverity, vulnerabilities[0].Severity)
assert.Equal(t, `Xerces-C is a validating XML parser written in a portable subset of C++. A flaw was found in the way the Xerces-C XML parser processed certain XML documents. A remote attacker could provide specially crafted XML input that, when parsed by an application using Xerces-C, would cause that application to crash.`, vulnerabilities[0].Description)
expectedFeatureVersions := []database.FeatureVersion{
@ -83,7 +82,7 @@ func TestRHELParser(t *testing.T) {
if assert.Nil(t, err) && assert.Len(t, vulnerabilities, 1) {
assert.Equal(t, "RHSA-2015:1207", vulnerabilities[0].Name)
assert.Equal(t, "https://rhn.redhat.com/errata/RHSA-2015-1207.html", vulnerabilities[0].Link)
assert.Equal(t, types.Critical, vulnerabilities[0].Severity)
assert.Equal(t, database.CriticalSeverity, vulnerabilities[0].Severity)
assert.Equal(t, `Mozilla Firefox is an open source web browser. XULRunner provides the XUL Runtime environment for Mozilla Firefox. Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.`, vulnerabilities[0].Description)
expectedFeatureVersions := []database.FeatureVersion{

0
updater/fetchers/rhel/testdata/fetcher_rhel_test.1.xml → ext/vulnsrc/rhel/testdata/fetcher_rhel_test.1.xml vendored
View File

0
updater/fetchers/rhel/testdata/fetcher_rhel_test.2.xml → ext/vulnsrc/rhel/testdata/fetcher_rhel_test.2.xml vendored
View File

0
updater/fetchers/ubuntu/testdata/fetcher_ubuntu_test.txt → ext/vulnsrc/ubuntu/testdata/fetcher_ubuntu_test.txt vendored
View File

111
updater/fetchers/ubuntu/ubuntu.go → ext/vulnsrc/ubuntu/ubuntu.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,16 +12,18 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package ubuntu implements a vulnerability source updater using the
// Ubuntu CVE Tracker.
package ubuntu
import (
"bufio"
"bytes"
"errors"
"fmt"
"io"
"io/ioutil"
"os"
"os/exec"
"regexp"
"strconv"
"strings"
@ -31,10 +33,8 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/ext/versionfmt/dpkg"
"github.com/coreos/clair/updater"
"github.com/coreos/clair/utils"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/utils/types"
"github.com/coreos/clair/ext/vulnsrc"
"github.com/coreos/clair/pkg/commonerr"
)
const (
@ -75,33 +75,27 @@ var (
affectsCaptureRegexp = regexp.MustCompile(`(?P<release>.*)_(?P<package>.*): (?P<status>[^\s]*)( \(+(?P<note>[^()]*)\)+)?`)
affectsCaptureRegexpNames = affectsCaptureRegexp.SubexpNames()
log = capnslog.NewPackageLogger("github.com/coreos/clair", "updater/fetchers/ubuntu")
// ErrFilesystem is returned when a fetcher fails to interact with the local filesystem.
ErrFilesystem = errors.New("updater/fetchers: something went wrong when interacting with the fs")
log = capnslog.NewPackageLogger("github.com/coreos/clair", "ext/vulnsrc/ubuntu")
)
// UbuntuFetcher implements updater.Fetcher and gets vulnerability updates from
// the Ubuntu CVE Tracker.
type UbuntuFetcher struct {
type updater struct {
repositoryLocalPath string
}
func init() {
updater.RegisterFetcher("Ubuntu", &UbuntuFetcher{})
vulnsrc.RegisterUpdater("ubuntu", &updater{})
}
// FetchUpdate gets vulnerability updates from the Ubuntu CVE Tracker.
func (fetcher *UbuntuFetcher) FetchUpdate(datastore database.Datastore) (resp updater.FetcherResponse, err error) {
func (u *updater) Update(datastore database.Datastore) (resp vulnsrc.UpdateResponse, err error) {
log.Info("fetching Ubuntu vulnerabilities")
// Pull the bzr repository.
if err = fetcher.pullRepository(); err != nil {
if err = u.pullRepository(); err != nil {
return resp, err
}
// Get revision number.
revisionNumber, err := getRevisionNumber(fetcher.repositoryLocalPath)
revisionNumber, err := getRevisionNumber(u.repositoryLocalPath)
if err != nil {
return resp, err
}
@ -113,7 +107,7 @@ func (fetcher *UbuntuFetcher) FetchUpdate(datastore database.Datastore) (resp up
}
// Get the list of vulnerabilities that we have to update.
modifiedCVE, err := collectModifiedVulnerabilities(revisionNumber, dbRevisionNumber, fetcher.repositoryLocalPath)
modifiedCVE, err := collectModifiedVulnerabilities(revisionNumber, dbRevisionNumber, u.repositoryLocalPath)
if err != nil {
return resp, err
}
@ -121,7 +115,7 @@ func (fetcher *UbuntuFetcher) FetchUpdate(datastore database.Datastore) (resp up
notes := make(map[string]struct{})
for cvePath := range modifiedCVE {
// Open the CVE file.
file, err := os.Open(fetcher.repositoryLocalPath + "/" + cvePath)
file, err := os.Open(u.repositoryLocalPath + "/" + cvePath)
if err != nil {
// This can happen when a file is modified and then moved in another
// commit.
@ -166,45 +160,57 @@ func (fetcher *UbuntuFetcher) FetchUpdate(datastore database.Datastore) (resp up
return
}
func (fetcher *UbuntuFetcher) pullRepository() (err error) {
func (u *updater) Clean() {
os.RemoveAll(u.repositoryLocalPath)
}
func (u *updater) pullRepository() (err error) {
// Determine whether we should branch or pull.
if _, pathExists := os.Stat(fetcher.repositoryLocalPath); fetcher.repositoryLocalPath == "" || os.IsNotExist(pathExists) {
if _, pathExists := os.Stat(u.repositoryLocalPath); u.repositoryLocalPath == "" || os.IsNotExist(pathExists) {
// Create a temporary folder to store the repository.
if fetcher.repositoryLocalPath, err = ioutil.TempDir(os.TempDir(), "ubuntu-cve-tracker"); err != nil {
return ErrFilesystem
if u.repositoryLocalPath, err = ioutil.TempDir(os.TempDir(), "ubuntu-cve-tracker"); err != nil {
return vulnsrc.ErrFilesystem
}
// Branch repository.
if out, err := utils.Exec(fetcher.repositoryLocalPath, "bzr", "branch", "--use-existing-dir", trackerRepository, "."); err != nil {
cmd := exec.Command("bzr", "branch", "--use-existing-dir", trackerRepository, ".")
cmd.Dir = u.repositoryLocalPath
if out, err := cmd.CombinedOutput(); err != nil {
log.Errorf("could not branch Ubuntu repository: %s. output: %s", err, out)
return cerrors.ErrCouldNotDownload
return commonerr.ErrCouldNotDownload
}
return nil
}
// Pull repository.
if out, err := utils.Exec(fetcher.repositoryLocalPath, "bzr", "pull", "--overwrite"); err != nil {
os.RemoveAll(fetcher.repositoryLocalPath)
cmd := exec.Command("bzr", "pull", "--overwrite")
cmd.Dir = u.repositoryLocalPath
if out, err := cmd.CombinedOutput(); err != nil {
os.RemoveAll(u.repositoryLocalPath)
log.Errorf("could not pull Ubuntu repository: %s. output: %s", err, out)
return cerrors.ErrCouldNotDownload
return commonerr.ErrCouldNotDownload
}
return nil
}
func getRevisionNumber(pathToRepo string) (int, error) {
out, err := utils.Exec(pathToRepo, "bzr", "revno")
cmd := exec.Command("bzr", "revno")
cmd.Dir = pathToRepo
out, err := cmd.CombinedOutput()
if err != nil {
log.Errorf("could not get Ubuntu repository's revision number: %s. output: %s", err, out)
return 0, cerrors.ErrCouldNotDownload
return 0, commonerr.ErrCouldNotDownload
}
revno, err := strconv.Atoi(strings.TrimSpace(string(out)))
if err != nil {
log.Errorf("could not parse Ubuntu repository's revision number: %s. output: %s", err, out)
return 0, cerrors.ErrCouldNotDownload
return 0, commonerr.ErrCouldNotDownload
}
return revno, nil
}
@ -217,14 +223,14 @@ func collectModifiedVulnerabilities(revision int, dbRevision, repositoryLocalPat
d, err := os.Open(repositoryLocalPath + "/" + folder)
if err != nil {
log.Errorf("could not open Ubuntu vulnerabilities repository's folder: %s", err)
return nil, ErrFilesystem
return nil, vulnsrc.ErrFilesystem
}
// Get the FileInfo of all the files in the directory.
names, err := d.Readdirnames(-1)
if err != nil {
log.Errorf("could not read Ubuntu vulnerabilities repository's folder:: %s.", err)
return nil, ErrFilesystem
return nil, vulnsrc.ErrFilesystem
}
// Add the vulnerabilities to the list.
@ -253,10 +259,12 @@ func collectModifiedVulnerabilities(revision int, dbRevision, repositoryLocalPat
}
// Handle a database that needs upgrading.
out, err := utils.Exec(repositoryLocalPath, "bzr", "log", "--verbose", "-r"+strconv.Itoa(dbRevisionInt+1)+"..", "-n0")
cmd := exec.Command("bzr", "log", "--verbose", "-r"+strconv.Itoa(dbRevisionInt+1)+"..", "-n0")
cmd.Dir = repositoryLocalPath
out, err := cmd.CombinedOutput()
if err != nil {
log.Errorf("could not get Ubuntu vulnerabilities repository logs: %s. output: %s", err, out)
return nil, cerrors.ErrCouldNotDownload
return nil, commonerr.ErrCouldNotDownload
}
scanner := bufio.NewScanner(bytes.NewReader(out))
@ -302,7 +310,7 @@ func parseUbuntuCVE(fileContent io.Reader) (vulnerability database.Vulnerability
priority = priority[:strings.Index(priority, " ")]
}
vulnerability.Severity = ubuntuPriorityToSeverity(priority)
vulnerability.Severity = SeverityFromPriority(priority)
continue
}
@ -389,33 +397,30 @@ func parseUbuntuCVE(fileContent io.Reader) (vulnerability database.Vulnerability
// If no priority has been provided (CVE-2007-0667 for instance), set the priority to Unknown
if vulnerability.Severity == "" {
vulnerability.Severity = types.Unknown
vulnerability.Severity = database.UnknownSeverity
}
return
}
func ubuntuPriorityToSeverity(priority string) types.Priority {
// SeverityFromPriority converts an priority from the Ubuntu CVE Tracker into
// a database.Severity.
func SeverityFromPriority(priority string) database.Severity {
switch priority {
case "untriaged":
return types.Unknown
return database.UnknownSeverity
case "negligible":
return types.Negligible
return database.NegligibleSeverity
case "low":
return types.Low
return database.LowSeverity
case "medium":
return types.Medium
return database.MediumSeverity
case "high":
return types.High
return database.HighSeverity
case "critical":
return types.Critical
return database.CriticalSeverity
default:
log.Warning("could not determine a vulnerability severity from: %s", priority)
return database.UnknownSeverity
}
log.Warning("Could not determine a vulnerability priority from: %s", priority)
return types.Unknown
}
// Clean deletes any allocated resources.
func (fetcher *UbuntuFetcher) Clean() {
os.RemoveAll(fetcher.repositoryLocalPath)
}

5
updater/fetchers/ubuntu/ubuntu_test.go → ext/vulnsrc/ubuntu/ubuntu_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -24,7 +24,6 @@ import (
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/versionfmt"
"github.com/coreos/clair/utils/types"
)
func TestUbuntuParser(t *testing.T) {
@ -37,7 +36,7 @@ func TestUbuntuParser(t *testing.T) {
vulnerability, unknownReleases, err := parseUbuntuCVE(testData)
if assert.Nil(t, err) {
assert.Equal(t, "CVE-2015-4471", vulnerability.Name)
assert.Equal(t, types.Medium, vulnerability.Severity)
assert.Equal(t, database.MediumSeverity, vulnerability.Severity)
assert.Equal(t, "Off-by-one error in the lzxd_decompress function in lzxd.c in libmspack before 0.5 allows remote attackers to cause a denial of service (buffer under-read and application crash) via a crafted CAB archive.", vulnerability.Description)
// Unknown release (line 28)

103
notifier/notifier.go → notifier.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,36 +12,29 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package notifier fetches notifications from the database and informs the specified remote handler
// about their existences, inviting the third party to actively query the API about it.
package notifier
package clair
import (
"time"
"github.com/coreos/pkg/capnslog"
"github.com/coreos/pkg/timeutil"
"github.com/pborman/uuid"
"github.com/prometheus/client_golang/prometheus"
"github.com/coreos/clair/config"
"github.com/coreos/clair/database"
"github.com/coreos/clair/utils"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/ext/notification"
"github.com/coreos/clair/pkg/commonerr"
"github.com/coreos/clair/pkg/stopper"
)
const (
checkInterval = 5 * time.Minute
refreshLockDuration = time.Minute * 2
lockDuration = time.Minute*8 + refreshLockDuration
maxBackOff = 15 * time.Minute
notifierCheckInterval = 5 * time.Minute
notifierMaxBackOff = 15 * time.Minute
notifierLockRefreshDuration = time.Minute * 2
notifierLockDuration = time.Minute*8 + notifierLockRefreshDuration
)
var (
log = capnslog.NewPackageLogger("github.com/coreos/clair", "notifier")
notifiers = make(map[string]Notifier)
promNotifierLatencyMilliseconds = prometheus.NewHistogram(prometheus.HistogramOpts{
Name: "clair_notifier_latency_milliseconds",
Help: "Time it takes to send a notification after it's been created.",
@ -53,57 +46,30 @@ var (
}, []string{"backend"})
)
// Notifier represents anything that can transmit notifications.
type Notifier interface {
// Configure attempts to initialize the notifier with the provided configuration.
// It returns whether the notifier is enabled or not.
Configure(*config.NotifierConfig) (bool, error)
// Send informs the existence of the specified notification.
Send(notification database.VulnerabilityNotification) error
}
func init() {
prometheus.MustRegister(promNotifierLatencyMilliseconds)
prometheus.MustRegister(promNotifierBackendErrorsTotal)
}
// RegisterNotifier makes a Fetcher available by the provided name.
// If Register is called twice with the same name or if driver is nil,
// it panics.
func RegisterNotifier(name string, n Notifier) {
if name == "" {
panic("notifier: could not register a Notifier with an empty name")
}
if n == nil {
panic("notifier: could not register a nil Notifier")
}
if _, dup := notifiers[name]; dup {
panic("notifier: RegisterNotifier called twice for " + name)
}
notifiers[name] = n
}
// Run starts the Notifier service.
func Run(config *config.NotifierConfig, datastore database.Datastore, stopper *utils.Stopper) {
// RunNotifier begins a process that checks for new notifications that should
// be sent out to third parties.
func RunNotifier(config *notification.Config, datastore database.Datastore, stopper *stopper.Stopper) {
defer stopper.End()
// Configure registered notifiers.
for notifierName, notifier := range notifiers {
if configured, err := notifier.Configure(config); configured {
log.Infof("notifier '%s' configured\n", notifierName)
for senderName, sender := range notification.Senders() {
if configured, err := sender.Configure(config); configured {
log.Infof("sender '%s' configured\n", senderName)
} else {
delete(notifiers, notifierName)
notification.UnregisterSender(senderName)
if err != nil {
log.Errorf("could not configure notifier '%s': %s", notifierName, err)
log.Errorf("could not configure notifier '%s': %s", senderName, err)
}
}
}
// Do not run the updater if there is no notifier enabled.
if len(notifiers) == 0 {
if len(notification.Senders()) == 0 {
log.Infof("notifier service is disabled")
return
}
@ -124,8 +90,9 @@ func Run(config *config.NotifierConfig, datastore database.Datastore, stopper *u
go func() {
success, interrupted := handleTask(*notification, stopper, config.Attempts)
if success {
utils.PrometheusObserveTimeMilliseconds(promNotifierLatencyMilliseconds, notification.Created)
datastore.SetNotificationNotified(notification.Name)
promNotifierLatencyMilliseconds.Observe(float64(time.Since(notification.Created).Nanoseconds()) / float64(time.Millisecond))
}
if interrupted {
running = false
@ -140,8 +107,8 @@ func Run(config *config.NotifierConfig, datastore database.Datastore, stopper *u
select {
case <-done:
break outer
case <-time.After(refreshLockDuration):
datastore.Lock(notification.Name, whoAmI, lockDuration, true)
case <-time.After(notifierLockRefreshDuration):
datastore.Lock(notification.Name, whoAmI, notifierLockDuration, true)
}
}
}
@ -149,18 +116,18 @@ func Run(config *config.NotifierConfig, datastore database.Datastore, stopper *u
log.Info("notifier service stopped")
}
func findTask(datastore database.Datastore, renotifyInterval time.Duration, whoAmI string, stopper *utils.Stopper) *database.VulnerabilityNotification {
func findTask(datastore database.Datastore, renotifyInterval time.Duration, whoAmI string, stopper *stopper.Stopper) *database.VulnerabilityNotification {
for {
// Find a notification to send.
notification, err := datastore.GetAvailableNotification(renotifyInterval)
if err != nil {
// There is no notification or an error occurred.
if err != cerrors.ErrNotFound {
if err != commonerr.ErrNotFound {
log.Warningf("could not get notification to send: %s", err)
}
// Wait.
if !stopper.Sleep(checkInterval) {
if !stopper.Sleep(notifierCheckInterval) {
return nil
}
@ -168,39 +135,39 @@ func findTask(datastore database.Datastore, renotifyInterval time.Duration, whoA
}
// Lock the notification.
if hasLock, _ := datastore.Lock(notification.Name, whoAmI, lockDuration, false); hasLock {
if hasLock, _ := datastore.Lock(notification.Name, whoAmI, notifierLockDuration, false); hasLock {
log.Infof("found and locked a notification: %s", notification.Name)
return &notification
}
}
}
func handleTask(notification database.VulnerabilityNotification, st *utils.Stopper, maxAttempts int) (bool, bool) {
func handleTask(n database.VulnerabilityNotification, st *stopper.Stopper, maxAttempts int) (bool, bool) {
// Send notification.
for notifierName, notifier := range notifiers {
for senderName, sender := range notification.Senders() {
var attempts int
var backOff time.Duration
for {
// Max attempts exceeded.
if attempts >= maxAttempts {
log.Infof("giving up on sending notification '%s' via notifier '%s': max attempts exceeded (%d)\n", notification.Name, notifierName, maxAttempts)
log.Infof("giving up on sending notification '%s' via sender '%s': max attempts exceeded (%d)\n", n.Name, senderName, maxAttempts)
return false, false
}
// Backoff.
if backOff > 0 {
log.Infof("waiting %v before retrying to send notification '%s' via notifier '%s' (Attempt %d / %d)\n", backOff, notification.Name, notifierName, attempts+1, maxAttempts)
log.Infof("waiting %v before retrying to send notification '%s' via sender '%s' (Attempt %d / %d)\n", backOff, n.Name, senderName, attempts+1, maxAttempts)
if !st.Sleep(backOff) {
return false, true
}
}
// Send using the current notifier.
if err := notifier.Send(notification); err != nil {
if err := sender.Send(n); err != nil {
// Send failed; increase attempts/backoff and retry.
promNotifierBackendErrorsTotal.WithLabelValues(notifierName).Inc()
log.Errorf("could not send notification '%s' via notifier '%s': %v", notification.Name, notifierName, err)
backOff = timeutil.ExpBackoff(backOff, maxBackOff)
promNotifierBackendErrorsTotal.WithLabelValues(senderName).Inc()
log.Errorf("could not send notification '%s' via notifier '%s': %v", n.Name, senderName, err)
backOff = timeutil.ExpBackoff(backOff, notifierMaxBackOff)
attempts++
continue
}
@ -210,6 +177,6 @@ func handleTask(notification database.VulnerabilityNotification, st *utils.Stopp
}
}
log.Infof("successfully sent notification '%s'\n", notification.Name)
log.Infof("successfully sent notification '%s'\n", n.Name)
return true, false
}

7
utils/errors/errors.go → pkg/commonerr/errors.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,8 +12,9 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package errors defines error types that are used in several modules
package errors
// Package commonerr defines reusable error types common throughout the Clair
// codebase.
package commonerr
import "errors"

4
utils/stopper.go → pkg/stopper/stopper.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
package utils
package stopper
import (
"sync"

141
utils/tar.go → pkg/tarutil/tarutil.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,7 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
package utils
// Package tarutil implements some tar utility functions.
package tarutil
import (
"archive/tar"
@ -29,28 +30,87 @@ import (
var (
// ErrCouldNotExtract occurs when an extraction fails.
ErrCouldNotExtract = errors.New("utils: could not extract the archive")
ErrCouldNotExtract = errors.New("tarutil: could not extract the archive")
// ErrExtractedFileTooBig occurs when a file to extract is too big.
ErrExtractedFileTooBig = errors.New("utils: could not extract one or more files from the archive: file too big")
ErrExtractedFileTooBig = errors.New("tarutil: could not extract one or more files from the archive: file too big")
// MaxExtractableFileSize enforces the maximum size of a single file within a
// tarball that will be extracted. This protects against malicious files that
// may used in an attempt to perform a Denial of Service attack.
MaxExtractableFileSize int64 = 200 * 1024 * 1024 // 200 MiB
readLen = 6 // max bytes to sniff
gzipHeader = []byte{0x1f, 0x8b}
bzip2Header = []byte{0x42, 0x5a, 0x68}
xzHeader = []byte{0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00}
)
// XzReader is an io.ReadCloser which decompresses xz compressed data.
// FilesMap is a map of files' paths to their contents.
type FilesMap map[string][]byte
// ExtractFiles decompresses and extracts only the specified files from an
// io.Reader representing an archive.
func ExtractFiles(r io.Reader, filenames []string) (FilesMap, error) {
data := make(map[string][]byte)
// Decompress the archive.
tr, err := NewTarReadCloser(r)
if err != nil {
return data, ErrCouldNotExtract
}
defer tr.Close()
// For each element in the archive
for {
hdr, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
return data, ErrCouldNotExtract
}
// Get element filename
filename := hdr.Name
filename = strings.TrimPrefix(filename, "./")
// Determine if we should extract the element
toBeExtracted := false
for _, s := range filenames {
if strings.HasPrefix(filename, s) {
toBeExtracted = true
break
}
}
if toBeExtracted {
// File size limit
if hdr.Size > MaxExtractableFileSize {
return data, ErrExtractedFileTooBig
}
// Extract the element
if hdr.Typeflag == tar.TypeSymlink || hdr.Typeflag == tar.TypeLink || hdr.Typeflag == tar.TypeReg {
d, _ := ioutil.ReadAll(tr)
data[filename] = d
}
}
}
return data, nil
}
// XzReader implements io.ReadCloser for data compressed via `xz`.
type XzReader struct {
io.ReadCloser
cmd *exec.Cmd
closech chan error
}
// NewXzReader shells out to a command line xz executable (if
// available) to decompress the given io.Reader using the xz
// compression format and returns an *XzReader.
// NewXzReader returns an io.ReadCloser by executing a command line `xz`
// executable to decompress the provided io.Reader.
//
// It is the caller's responsibility to call Close on the XzReader when done.
func NewXzReader(r io.Reader) (*XzReader, error) {
rpipe, wpipe := io.Pipe()
@ -74,6 +134,7 @@ func NewXzReader(r io.Reader) (*XzReader, error) {
return &XzReader{rpipe, cmd, closech}, nil
}
// Close cleans up the resources used by an XzReader.
func (r *XzReader) Close() error {
r.ReadCloser.Close()
r.cmd.Process.Kill()
@ -88,72 +149,20 @@ type TarReadCloser struct {
io.Closer
}
// Close cleans up the resources used by a TarReadCloser.
func (r *TarReadCloser) Close() error {
return r.Closer.Close()
}
// SelectivelyExtractArchive extracts the specified files and folders
// from targz data read from the given reader and store them in a map indexed by file paths
func SelectivelyExtractArchive(r io.Reader, prefix string, toExtract []string, maxFileSize int64) (map[string][]byte, error) {
data := make(map[string][]byte)
// Create a tar or tar/tar-gzip/tar-bzip2/tar-xz reader
tr, err := getTarReader(r)
if err != nil {
return data, ErrCouldNotExtract
}
defer tr.Close()
// For each element in the archive
for {
hdr, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
return data, ErrCouldNotExtract
}
// Get element filename
filename := hdr.Name
filename = strings.TrimPrefix(filename, "./")
if prefix != "" {
filename = strings.TrimPrefix(filename, prefix)
}
// Determine if we should extract the element
toBeExtracted := false
for _, s := range toExtract {
if strings.HasPrefix(filename, s) {
toBeExtracted = true
break
}
}
if toBeExtracted {
// File size limit
if maxFileSize > 0 && hdr.Size > maxFileSize {
return data, ErrExtractedFileTooBig
}
// Extract the element
if hdr.Typeflag == tar.TypeSymlink || hdr.Typeflag == tar.TypeLink || hdr.Typeflag == tar.TypeReg {
d, _ := ioutil.ReadAll(tr)
data[filename] = d
}
}
}
return data, nil
}
// getTarReader returns a TarReaderCloser associated with the specified io.Reader.
// NewTarReadCloser attempts to detect the compression algorithm for an
// io.Reader and returns a TarReadCloser wrapping the Reader to transparently
// decompress the contents.
//
// Gzip/Bzip2/XZ detection is done by using the magic numbers:
// Gzip: the first two bytes should be 0x1f and 0x8b. Defined in the RFC1952.
// Bzip2: the first three bytes should be 0x42, 0x5a and 0x68. No RFC.
// XZ: the first three bytes should be 0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00. No RFC.
func getTarReader(r io.Reader) (*TarReadCloser, error) {
func NewTarReadCloser(r io.Reader) (*TarReadCloser, error) {
br := bufio.NewReader(r)
header, err := br.Peek(readLen)
if err == nil {

80
pkg/tarutil/tarutil_test.go Normal file
View File

@ -0,0 +1,80 @@
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package tarutil
import (
"bytes"
"os"
"path/filepath"
"runtime"
"testing"
"github.com/stretchr/testify/assert"
)
var testTarballs = []string{
"utils_test.tar",
"utils_test.tar.gz",
"utils_test.tar.bz2",
"utils_test.tar.xz",
}
func testfilepath(filename string) string {
_, path, _, _ := runtime.Caller(0)
testDataDir := "/testdata"
return filepath.Join(filepath.Dir(path), testDataDir, filename)
}
func TestExtract(t *testing.T) {
for _, filename := range testTarballs {
f, err := os.Open(testfilepath(filename))
assert.Nil(t, err)
defer f.Close()
data, err := ExtractFiles(f, []string{"test/"})
assert.Nil(t, err)
if c, n := data["test/test.txt"]; !n {
assert.Fail(t, "test/test.txt should have been extracted")
} else {
assert.NotEqual(t, 0, len(c) > 0, "test/test.txt file is empty")
}
if _, n := data["test.txt"]; n {
assert.Fail(t, "test.txt should not be extracted")
}
}
}
func TestExtractUncompressedData(t *testing.T) {
for _, filename := range testTarballs {
f, err := os.Open(testfilepath(filename))
assert.Nil(t, err)
defer f.Close()
_, err = ExtractFiles(bytes.NewReader([]byte("that string does not represent a tar or tar-gzip file")), []string{})
assert.Error(t, err, "Extracting uncompressed data should return an error")
}
}
func TestMaxExtractableFileSize(t *testing.T) {
for _, filename := range testTarballs {
f, err := os.Open(testfilepath(filename))
assert.Nil(t, err)
defer f.Close()
MaxExtractableFileSize = 50
_, err = ExtractFiles(f, []string{"test"})
assert.Equal(t, ErrExtractedFileTooBig, err)
}
}

0
utils/testdata/utils_test.tar → pkg/tarutil/testdata/utils_test.tar vendored
View File

0
utils/testdata/utils_test.tar.bz2 → pkg/tarutil/testdata/utils_test.tar.bz2 vendored
View File

0
utils/testdata/utils_test.tar.gz → pkg/tarutil/testdata/utils_test.tar.gz vendored
View File

0
utils/testdata/utils_test.tar.xz → pkg/tarutil/testdata/utils_test.tar.xz vendored
View File

0
worker/testdata/DistUpgrade/blank.tar.gz → testdata/DistUpgrade/blank.tar.gz vendored
View File

0
worker/testdata/DistUpgrade/jessie.tar.gz → testdata/DistUpgrade/jessie.tar.gz vendored
View File

0
worker/testdata/DistUpgrade/wheezy.tar.gz → testdata/DistUpgrade/wheezy.tar.gz vendored
View File

130
updater/updater.go → updater.go
View File

@ -1,4 +1,4 @@
// Copyright 2015 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,9 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// Package updater updates the vulnerability database periodically using
// the registered vulnerability fetchers.
package updater
package clair
import (
"math/rand"
@ -22,25 +20,25 @@ import (
"sync"
"time"
"github.com/coreos/clair/config"
"github.com/coreos/clair/database"
"github.com/coreos/clair/utils"
"github.com/coreos/pkg/capnslog"
"github.com/pborman/uuid"
"github.com/prometheus/client_golang/prometheus"
"github.com/coreos/clair/database"
"github.com/coreos/clair/ext/vulnmdsrc"
"github.com/coreos/clair/ext/vulnsrc"
"github.com/coreos/clair/pkg/stopper"
)
const (
flagName = "updater/last"
notesFlagName = "updater/notes"
lockName = "updater"
lockDuration = refreshLockDuration + time.Minute*2
refreshLockDuration = time.Minute * 8
updaterLastFlagName = "updater/last"
updaterLockName = "updater"
updaterLockDuration = updaterLockRefreshDuration + time.Minute*2
updaterLockRefreshDuration = time.Minute * 8
)
var (
log = capnslog.NewPackageLogger("github.com/coreos/clair", "updater")
log = capnslog.NewPackageLogger("github.com/coreos/clair", "clair")
promUpdaterErrorsTotal = prometheus.NewCounter(prometheus.CounterOpts{
Name: "clair_updater_errors_total",
@ -64,8 +62,14 @@ func init() {
prometheus.MustRegister(promUpdaterNotesTotal)
}
// Run updates the vulnerability database at regular intervals.
func Run(config *config.UpdaterConfig, datastore database.Datastore, st *utils.Stopper) {
// UpdaterConfig is the configuration for the Updater service.
type UpdaterConfig struct {
Interval time.Duration
}
// RunUpdater begins a process that updates the vulnerability database at
// regular intervals.
func RunUpdater(config *UpdaterConfig, datastore database.Datastore, st *stopper.Stopper) {
defer st.End()
// Do not run the updater if there is no config or if the interval is 0.
@ -95,12 +99,12 @@ func Run(config *config.UpdaterConfig, datastore database.Datastore, st *utils.S
if nextUpdate.Before(time.Now().UTC()) {
// Attempt to get a lock on the the update.
log.Debug("attempting to obtain update lock")
hasLock, hasLockUntil := datastore.Lock(lockName, whoAmI, lockDuration, false)
hasLock, hasLockUntil := datastore.Lock(updaterLockName, whoAmI, updaterLockDuration, false)
if hasLock {
// Launch update in a new go routine.
doneC := make(chan bool, 1)
go func() {
Update(datastore, firstUpdate)
update(datastore, firstUpdate)
doneC <- true
}()
@ -108,23 +112,23 @@ func Run(config *config.UpdaterConfig, datastore database.Datastore, st *utils.S
select {
case <-doneC:
done = true
case <-time.After(refreshLockDuration):
case <-time.After(updaterLockRefreshDuration):
// Refresh the lock until the update is done.
datastore.Lock(lockName, whoAmI, lockDuration, true)
datastore.Lock(updaterLockName, whoAmI, updaterLockDuration, true)
case <-st.Chan():
stop = true
}
}
// Unlock the update.
datastore.Unlock(lockName, whoAmI)
datastore.Unlock(updaterLockName, whoAmI)
if stop {
break
}
continue
} else {
lockOwner, lockExpiration, err := datastore.FindLock(lockName)
lockOwner, lockExpiration, err := datastore.FindLock(updaterLockName)
if err != nil {
log.Debug("update lock is already taken")
nextUpdate = hasLockUntil
@ -147,19 +151,19 @@ func Run(config *config.UpdaterConfig, datastore database.Datastore, st *utils.S
}
// Clean resources.
for _, metadataFetcher := range metadataFetchers {
metadataFetcher.Clean()
for _, appenders := range vulnmdsrc.Appenders() {
appenders.Clean()
}
for _, fetcher := range fetchers {
fetcher.Clean()
for _, updaters := range vulnsrc.Updaters() {
updaters.Clean()
}
log.Info("updater service stopped")
}
// Update fetches all the vulnerabilities from the registered fetchers, upserts
// update fetches all the vulnerabilities from the registered fetchers, upserts
// them into the database and then sends notifications.
func Update(datastore database.Datastore, firstUpdate bool) {
func update(datastore database.Datastore, firstUpdate bool) {
defer setUpdaterDuration(time.Now())
log.Info("updating vulnerabilities")
@ -190,7 +194,7 @@ func Update(datastore database.Datastore, firstUpdate bool) {
// Update last successful update if every fetchers worked properly.
if status {
datastore.InsertKeyValue(flagName, strconv.FormatInt(time.Now().UTC().Unix(), 10))
datastore.InsertKeyValue(updaterLastFlagName, strconv.FormatInt(time.Now().UTC().Unix(), 10))
}
log.Info("update finished")
@ -209,10 +213,10 @@ func fetch(datastore database.Datastore) (bool, []database.Vulnerability, map[st
// Fetch updates in parallel.
log.Info("fetching vulnerability updates")
var responseC = make(chan *FetcherResponse, 0)
for n, f := range fetchers {
go func(name string, fetcher Fetcher) {
response, err := fetcher.FetchUpdate(datastore)
var responseC = make(chan *vulnsrc.UpdateResponse, 0)
for n, u := range vulnsrc.Updaters() {
go func(name string, u vulnsrc.Updater) {
response, err := u.Update(datastore)
if err != nil {
promUpdaterErrorsTotal.Inc()
log.Errorf("an error occured when fetching update '%s': %s.", name, err)
@ -222,11 +226,11 @@ func fetch(datastore database.Datastore) (bool, []database.Vulnerability, map[st
}
responseC <- &response
}(n, f)
}(n, u)
}
// Collect results of updates.
for i := 0; i < len(fetchers); i++ {
for i := 0; i < len(vulnsrc.Updaters()); i++ {
resp := <-responseC
if resp != nil {
vulnerabilities = append(vulnerabilities, doVulnerabilitiesNamespacing(resp.Vulnerabilities)...)
@ -243,42 +247,43 @@ func fetch(datastore database.Datastore) (bool, []database.Vulnerability, map[st
// Add metadata to the specified vulnerabilities using the registered MetadataFetchers, in parallel.
func addMetadata(datastore database.Datastore, vulnerabilities []database.Vulnerability) []database.Vulnerability {
if len(metadataFetchers) == 0 {
if len(vulnmdsrc.Appenders()) == 0 {
return vulnerabilities
}
log.Info("adding metadata to vulnerabilities")
// Wrap vulnerabilities in VulnerabilityWithLock.
// It ensures that only one metadata fetcher at a time can modify the Metadata map.
vulnerabilitiesWithLocks := make([]*VulnerabilityWithLock, 0, len(vulnerabilities))
// Add a mutex to each vulnerability to ensure that only one appender at a
// time can modify the vulnerability's Metadata map.
lockableVulnerabilities := make([]*lockableVulnerability, 0, len(vulnerabilities))
for i := 0; i < len(vulnerabilities); i++ {
vulnerabilitiesWithLocks = append(vulnerabilitiesWithLocks, &VulnerabilityWithLock{
lockableVulnerabilities = append(lockableVulnerabilities, &lockableVulnerability{
Vulnerability: &vulnerabilities[i],
})
}
var wg sync.WaitGroup
wg.Add(len(metadataFetchers))
wg.Add(len(vulnmdsrc.Appenders()))
for n, f := range metadataFetchers {
go func(name string, metadataFetcher MetadataFetcher) {
for n, a := range vulnmdsrc.Appenders() {
go func(name string, appender vulnmdsrc.Appender) {
defer wg.Done()
// Load the metadata fetcher.
if err := metadataFetcher.Load(datastore); err != nil {
// Build up a metadata cache.
if err := appender.BuildCache(datastore); err != nil {
promUpdaterErrorsTotal.Inc()
log.Errorf("an error occured when loading metadata fetcher '%s': %s.", name, err)
return
}
// Add metadata to each vulnerability.
for _, vulnerability := range vulnerabilitiesWithLocks {
metadataFetcher.AddMetadata(vulnerability)
// Append vulnerability metadata to each vulnerability.
for _, vulnerability := range lockableVulnerabilities {
appender.Append(vulnerability.Name, vulnerability.appendFunc)
}
metadataFetcher.Unload()
}(n, f)
// Purge the metadata cache.
appender.PurgeCache()
}(n, a)
}
wg.Wait()
@ -287,7 +292,7 @@ func addMetadata(datastore database.Datastore, vulnerabilities []database.Vulner
}
func getLastUpdate(datastore database.Datastore) (time.Time, bool, error) {
lastUpdateTSS, err := datastore.GetKeyValue(flagName)
lastUpdateTSS, err := datastore.GetKeyValue(updaterLastFlagName)
if err != nil {
return time.Time{}, false, err
}
@ -305,6 +310,29 @@ func getLastUpdate(datastore database.Datastore) (time.Time, bool, error) {
return time.Unix(lastUpdateTS, 0).UTC(), false, nil
}
type lockableVulnerability struct {
*database.Vulnerability
sync.Mutex
}
func (lv *lockableVulnerability) appendFunc(metadataKey string, metadata interface{}, severity database.Severity) {
lv.Lock()
defer lv.Unlock()
// If necessary, initialize the metadata map for the vulnerability.
if lv.Metadata == nil {
lv.Metadata = make(map[string]interface{})
}
// Append the metadata.
lv.Metadata[metadataKey] = metadata
// If necessary, provide a severity for the vulnerability.
if lv.Severity == database.UnknownSeverity {
lv.Severity = severity
}
}
// doVulnerabilitiesNamespacing takes Vulnerabilities that don't have a Namespace and split them
// into multiple vulnerabilities that have a Namespace and only contains the FixedIn
// FeatureVersions corresponding to their Namespace.

56
updater/fetchers.go
View File

@ -1,56 +0,0 @@
// Copyright 2015 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package updater
import "github.com/coreos/clair/database"
var fetchers = make(map[string]Fetcher)
// Fetcher represents anything that can fetch vulnerabilities.
type Fetcher interface {
// FetchUpdate gets vulnerability updates.
FetchUpdate(database.Datastore) (FetcherResponse, error)
// Clean deletes any allocated resources.
// It is invoked when Clair stops.
Clean()
}
// FetcherResponse represents the sum of results of an update.
type FetcherResponse struct {
FlagName string
FlagValue string
Notes []string
Vulnerabilities []database.Vulnerability
}
// RegisterFetcher makes a Fetcher available by the provided name.
// If Register is called twice with the same name or if driver is nil,
// it panics.
func RegisterFetcher(name string, f Fetcher) {
if name == "" {
panic("updater: could not register a Fetcher with an empty name")
}
if f == nil {
panic("updater: could not register a nil Fetcher")
}
if _, dup := fetchers[name]; dup {
panic("updater: RegisterFetcher called twice for " + name)
}
fetchers[name] = f
}

64
updater/metadata_fetchers.go
View File

@ -1,64 +0,0 @@
// Copyright 2015 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package updater
import (
"sync"
"github.com/coreos/clair/database"
)
var metadataFetchers = make(map[string]MetadataFetcher)
type VulnerabilityWithLock struct {
*database.Vulnerability
Lock sync.Mutex
}
// MetadataFetcher
type MetadataFetcher interface {
// Load runs right before the Updater calls AddMetadata for each vulnerabilities.
Load(database.Datastore) error
// AddMetadata adds metadata to the given database.Vulnerability.
// It is expected that the fetcher uses .Lock.Lock() when manipulating the Metadata map.
AddMetadata(*VulnerabilityWithLock) error
// Unload runs right after the Updater finished calling AddMetadata for every vulnerabilities.
Unload()
// Clean deletes any allocated resources.
// It is invoked when Clair stops.
Clean()
}
// RegisterFetcher makes a Fetcher available by the provided name.
// If Register is called twice with the same name or if driver is nil,
// it panics.
func RegisterMetadataFetcher(name string, f MetadataFetcher) {
if name == "" {
panic("updater: could not register a MetadataFetcher with an empty name")
}
if f == nil {
panic("updater: could not register a nil MetadataFetcher")
}
if _, dup := fetchers[name]; dup {
panic("updater: RegisterMetadataFetcher called twice for " + name)
}
metadataFetchers[name] = f
}

19
updater/metadata_fetchers/nvd/nested_read_closer.go
View File

@ -1,19 +0,0 @@
package nvd
import "io"
// NestedReadCloser wraps an io.Reader and implements io.ReadCloser by closing every embed
// io.ReadCloser.
// It allows chaining io.ReadCloser together and still keep the ability to close them all in a
// simple manner.
type NestedReadCloser struct {
io.Reader
NestedReadClosers []io.ReadCloser
}
// Close closes the gzip.Reader and the underlying io.ReadCloser.
func (nrc *NestedReadCloser) Close() {
for _, nestedReadCloser := range nrc.NestedReadClosers {
nestedReadCloser.Close()
}
}

4
updater/updater_test.go → updater_test.go
View File

@ -1,4 +1,4 @@
// Copyright 2016 clair authors
// Copyright 2017 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
package updater
package clair
import (
"fmt"

77
utils/http/http.go
View File

@ -1,77 +0,0 @@
// Copyright 2015 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package http provides utility functions for HTTP servers and clients.
package http
import (
"encoding/json"
"io"
"net/http"
"github.com/coreos/clair/database"
"github.com/coreos/clair/utils"
cerrors "github.com/coreos/clair/utils/errors"
"github.com/coreos/clair/worker"
)
// MaxBodySize is the maximum number of bytes that ParseHTTPBody reads from an http.Request.Body.
const MaxBodySize int64 = 1048576
// WriteHTTP writes a JSON-encoded object to a http.ResponseWriter, as well as
// a HTTP status code.
func WriteHTTP(w http.ResponseWriter, httpStatus int, v interface{}) {
w.WriteHeader(httpStatus)
if v != nil {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
result, _ := json.Marshal(v)
w.Write(result)
}
}
// WriteHTTPError writes an error, wrapped in the Message field of a JSON-encoded
// object to a http.ResponseWriter, as well as a HTTP status code.
// If the status code is 0, handleError tries to guess the proper HTTP status
// code from the error type.
func WriteHTTPError(w http.ResponseWriter, httpStatus int, err error) {
if httpStatus == 0 {
httpStatus = http.StatusInternalServerError
// Try to guess the http status code from the error type
if _, isBadRequestError := err.(*cerrors.ErrBadRequest); isBadRequestError {
httpStatus = http.StatusBadRequest
} else {
switch err {
case cerrors.ErrNotFound:
httpStatus = http.StatusNotFound
case database.ErrBackendException:
httpStatus = http.StatusServiceUnavailable
case worker.ErrParentUnknown, worker.ErrUnsupported, utils.ErrCouldNotExtract, utils.ErrExtractedFileTooBig:
httpStatus = http.StatusBadRequest
}
}
}
WriteHTTP(w, httpStatus, struct{ Message string }{Message: err.Error()})
}
// ParseHTTPBody reads a JSON-encoded body from a http.Request and unmarshals it
// into the provided object.
func ParseHTTPBody(r *http.Request, v interface{}) (int, error) {
defer r.Body.Close()
err := json.NewDecoder(io.LimitReader(r.Body, MaxBodySize)).Decode(v)
if err != nil {
return http.StatusUnsupportedMediaType, err
}
return 0, nil
}

13
utils/prometheus.go
View File

@ -1,13 +0,0 @@
package utils
import (
"time"
"github.com/prometheus/client_golang/prometheus"
)
// PrometheusObserveTimeMilliseconds observes the elapsed time since start, in milliseconds,
// on the specified Prometheus Histogram.
func PrometheusObserveTimeMilliseconds(h prometheus.Histogram, start time.Time) {
h.Observe(float64(time.Since(start).Nanoseconds()) / float64(time.Millisecond))
}

75
utils/string.go
View File

@ -1,75 +0,0 @@
// Copyright 2015 clair authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package utils
import "regexp"
var urlParametersRegexp = regexp.MustCompile(`(\?|\&)([^=]+)\=([^ &]+)`)
// CleanURL removes all parameters from an URL.
func CleanURL(str string) string {
return urlParametersRegexp.ReplaceAllString(str, "")
}
// Contains looks for a string into an array of strings and returns whether
// the string exists.
func Contains(needle string, haystack []string) bool {
for _, h := range haystack {
if h == needle {
return true
}
}
return false
}
// CompareStringLists returns the strings that are present in X but not in Y.
func CompareStringLists(X, Y []string) []string {
m := make(map[string]bool)
for _, y := range Y {
m[y] = true
}
diff := []string{}
for _, x := range X {
if m[x] {
continue
}
diff = append(diff, x)
m[x] = true
}
return diff
}
// CompareStringListsInBoth returns the strings that are present in both X and Y.
func CompareStringListsInBoth(X, Y []string) []string {
m := make(map[string]struct{})
for _, y := range Y {
m[y] = struct{}{}
}
diff := []string{}
for _, x := range X {
if _, e := m[x]; e {
diff = append(diff, x)
delete(m, x)
}
}
return diff
}

Some files were not shown because too many files have changed in this diff Show More