Commit Graph

118 Commits

Author SHA1 Message Date
Sida Chen
a5c6400065 database: postgres implementation with tests. 2017-08-10 11:25:29 -04:00
Sida Chen
fb32dcfa58 Clair Logic, Extensions: updated mock tests, extensions, basic logic
Main Clair logic is changed in worker, updater, notifier for better adapting
ancestry schema. Extensions are updated with the new model and feature lister
 and namespace detector drivers are able to specify the specific listers and
detectors used to process layer's content. InRange and GetFixedIn interfaces
are added to Version format for adapting ranged affected features and next
available fixed in in the future. Tests for worker, updater and extensions
are fixed.
2017-08-10 11:24:40 -04:00
Sida Chen
57b146d0d8 Datastore: updated for Clair V3, decoupled interfaces and models 2017-08-10 11:22:44 -04:00
Tianon Gravi
de271820a8 Add Debian Buster (10) and update "*stable" aliases
See https://lists.debian.org/debian-announce/2017/msg00003.html for the
official release announcement for Debian Stretch.
2017-06-19 07:54:51 -07:00
Jimmy Zelinskie
abd7d2e013 Merge pull request #394 from KeyboardNerd/multiplens
added support for detecting multiple namespaces in a layer
2017-05-24 17:22:08 -04:00
Sida Chen
75d5d40d79 featurens: added multiple namespace testing for namespace detector 2017-05-24 17:18:11 -04:00
Sida Chen
bffa6499b7 added support for detect multiple namespaces in a layer
created table layer_namespace to store the many to many unique mapping of layers and namespaces
changed v1 api to provide a list of namespaces for each layer
changed namespace detector to use all registered detectors to detect namespaces
updated tests for multiple namespaces

Fixes #150
2017-05-24 17:01:51 -04:00
Jimmy Zelinskie
c2d8aec157 Merge pull request #382 from caipre/patch-1
pgsql: Change layer name column data type
2017-05-19 20:07:53 -04:00
Nick Platt
aea74550e1 pgsql: Expand layer, namespace column widths
Presently the layer and namespace tables use type `varchar(128)` for
their respective name columns. For layer, this width works fine enough
using the sha256 digests provided by docker. However, if one wishes to
encode the image name into the layer  name (eg, to avoid collisions like
in [0]), the limit of 128 bytes starts to feel a bit cramped. Bump to
256 bytes, since that "ought to be enough for anybody." (TM)

[0]: https://github.com/coreos/clair/issues/319
2017-05-19 18:22:04 -04:00
Jimmy Zelinskie
0305dde964 database/models: MetadataMap decodes from string
github.com/lib/pq began decoding text-like fields as strings to
Scanners.

See lib/pq@e2402a7cd1
2017-05-06 17:01:41 -04:00
Sida Chen
9306e99368 converted to structured logging by using logrus
changed from capnslog to logrus for logging JSON structured message.

finished issue #383
2017-05-04 13:59:57 -04:00
Tianon Gravi
b6ab5d8168 Add Ubuntu Zesty and Artful to UbuntuReleasesMapping
See also https://wiki.ubuntu.com/Releases
2017-04-26 14:45:32 -07:00
Jimmy Zelinskie
9e875f748d database/pgsql: copy whole namespace 2017-02-23 02:05:33 -05:00
Jimmy Zelinskie
6a569fd945 move config to main / decentralize config
This puts config in its relevant location and moves functions around
loading config files into the main package.

As a side effect of removing cyclic imports for the API config, the
context library is no longer used.
2017-01-27 00:36:13 -05:00
Jimmy Zelinskie
9c63a63944 clair: mv updater clair and mv severity to db 2017-01-22 23:20:56 -05:00
Jimmy Zelinskie
3e4dc3834f utils: remove string.go 2017-01-22 23:02:51 -05:00
Jimmy Zelinskie
e7f72ef5ad utils: rm prometheus.go 2017-01-22 23:02:51 -05:00
Jimmy Zelinskie
343e24eb7e clair: remove types package
This removes the `types` package instead moving the contents to the
top-level clair package.
This change also renames the `Priority` type to `Severity` in order to
reduce confusion.
This change also removes the IsValid method and replaces it with a safe
constructor to avoid the creation of invalid values.
Many docstrings were tweaked in the making of this commit.
2017-01-22 23:02:51 -05:00
Jimmy Zelinskie
78cef02fda pkg: cerrors -> commonerr 2017-01-22 23:02:50 -05:00
Jimmy Zelinskie
224ff82543 pgsql/migrations: fix dpkg default versionfmt 2017-01-20 14:42:33 -05:00
Jimmy Zelinskie
ca9f340a91 pgsql: only select distinct layers 2017-01-19 17:09:57 -05:00
Jimmy Zelinskie
ea73aa153d pgsql: searchNotificationLayerIntroducingVulnerability order by layer ID
This fixes a bug where the API was returning Notification pages ordered
by LDFV.ID instead of by Layer ID.
2017-01-19 13:45:07 -05:00
Jimmy Zelinskie
627b98ef31 db/pgsql/feature: fix SQL error reporting 2017-01-03 17:06:08 -05:00
Jimmy Zelinskie
8d29bf860d versionfmt: convert to using constant over literal 2017-01-03 16:00:20 -05:00
Jimmy Zelinskie
8df8170ba5 db/pgsql/migration: convert to pure SQL 2017-01-03 15:59:22 -05:00
Jimmy Zelinskie
9e39a26f26 backfill version_format column 2017-01-03 13:15:14 -05:00
Jimmy Zelinskie
033709eaea add registerable version formats
Since we only ever used dpkg, this change shims everything into using
dpkg.
2016-12-30 12:51:24 -05:00
Jimmy Zelinskie
9338f28e82 psql/migrations: fix ordering 2016-12-25 19:25:57 -05:00
Jimmy Zelinskie
d4522e9c6e api/v1: indexed layers for notifications
This change deprecates the old LayersIntroducingVulnerability for a new
one that orders output and contains an Index. This index is not
guaranteed to be consistent across multiple notifications, despite the
current Postgres implementation using the primary key of Layer table.
2016-12-06 19:23:33 -05:00
Quentin Machu
1fcae6abb8 Merge pull request #280 from coreos/add_idx_deleted_at
pgsql/migrations: add index on Vulnerability_Notification.deleted_at
2016-12-06 19:48:40 +01:00
Quentin Machu
7a3dd5c817 pgsql: Disable hashjoins to get introducing layers for notifications 2016-12-06 16:19:10 +01:00
Quentin Machu
eeb13a02ba pgsql/migrations: add index on Vulnerability_Notification.deleted_at
`searchNotificationAvailable` never effectively use any indexes because:
- `notified_at < $1`, where $1 is a recent timestamp, returns the
  majority of the table and therefore it is cheaper for PostgreSQL
  to use a sequential scan on the table.
- there is no index for `deleted_at IS NULL`.
However, when Clair has been running for long enough, the grand majority
of rows (99%+) are expected to have a non-NULL `deleted_at` field. This
commit adds a new index on this very field in order to fetch the
remaining 1% in the blink of an eye.

In other words, instead of realizing a full table scan for each
`searchNotificationAvailable` query, we'll use the small branch of a new
index, reducing the total cost from over 30k to a mere 150 on a Clair
database that has already managed more than 1 000 000 notifications.
2016-12-06 14:39:52 +01:00
Jimmy Zelinskie
dab6e492b8 Merge pull request #279 from coreos/searchintro_optimize
pgsql: Reduce cost of GetNotification by 2.5x
2016-12-04 12:08:50 -05:00
Quentin Machu
dc8f71024f pgsql: Reduce cost of GetNotification by 2.5
By delaying the Layer join to the very end, we can cut the query costs from 540,836 to 219,477.

See Pull Request for details.
2016-12-04 13:21:47 +01:00
Jimmy Zelinskie
7cff31a058 pgsql/migrations: add ldfv compound index
This speeds up the SearchNotificationLayerIntroducingVulnerability query
by an order magnitude.
2016-12-04 05:02:15 -05:00
Jimmy Zelinskie
9dc002621a psql: add useful indexes
This adds some missing UNIQUE constraints and indexes for the
vulnerability table that should improve query performance.
2016-12-02 15:48:12 -05:00
Quentin Machu
7fddd68f4f Merge pull request #263 from Quentin-M/rhel_unique_fixedin
pgsql: Do not insert entry in Vulnerability_FixedIn_Feature if existing
2016-11-13 13:25:08 +01:00
Quentin Machu
ec0aad9b7a pgsql: Use booleans instead of varchar to return creation status 2016-11-12 15:42:59 +01:00
Quentin Machu
cd23262e41 pgsql: Do not insert entry in Vulnerability_FixedIn_Feature if existing
Fixes #238
2016-11-11 19:17:32 +01:00
Quentin Machu
b8865b2106 pgsql: Replace liamstask/goose by remind101/migrate
Fixes #93
2016-11-11 18:11:48 +01:00
Matt Moore
90cc8243ba Add Ubuntu yakkety to the namespace mapping. 2016-10-24 21:08:02 -07:00
Quentin Machu
a03459d02e Merge pull request #165 from Quentin-M/db_registration
Allow specifying datastore driver by config, relocate upgrade detection, mock datastore
2016-05-20 12:20:26 -05:00
Jimmy Zelinskie
5d8336acb3 pgsql: use subquery to plan GetNotification query (#182)
This change enables the query planner to wait and sort the result set of
our query rather than attempting to re-use the layer table's index for
the ORDER BY clause. Because the result set is always small, this makes
queries that were previous tens of seconds, now tens of milliseconds.
2016-05-20 13:10:00 -04:00
Quentin Machu
836d37b275 *: use path/filepath instead of path 2016-05-20 12:01:31 -05:00
Quentin Machu
b99e2b50e2 database: Add some missing copyright headers 2016-05-20 12:01:31 -05:00
Quentin Machu
629d2ce662 database: Mock Datastore interface 2016-05-20 12:01:30 -05:00
Quentin Machu
a38fbf6cfe worker/database: Move upgrade detection logic out of database to worker 2016-05-20 12:01:30 -05:00
Jimmy Zelinskie
51f9c5dcb4 pgsql: remove unnecessary join used in GetNotification (#179) 2016-05-19 11:45:38 -04:00
Quentin Machu
e7b960c05b database: Allow specifying datastore driver by config
Fixes #145
2016-05-11 15:29:13 -07:00
Quentin Machu
79ba99bbea database: Fix invalid error message 2016-03-18 12:25:01 -04:00