Tianon Gravi
de271820a8
Add Debian Buster (10) and update "*stable" aliases
...
See https://lists.debian.org/debian-announce/2017/msg00003.html for the
official release announcement for Debian Stretch.
2017-06-19 07:54:51 -07:00
Jimmy Zelinskie
abd7d2e013
Merge pull request #394 from KeyboardNerd/multiplens
...
added support for detecting multiple namespaces in a layer
2017-05-24 17:22:08 -04:00
Sida Chen
75d5d40d79
featurens: added multiple namespace testing for namespace detector
2017-05-24 17:18:11 -04:00
Sida Chen
bffa6499b7
added support for detect multiple namespaces in a layer
...
created table layer_namespace to store the many to many unique mapping of layers and namespaces
changed v1 api to provide a list of namespaces for each layer
changed namespace detector to use all registered detectors to detect namespaces
updated tests for multiple namespaces
Fixes #150
2017-05-24 17:01:51 -04:00
Jimmy Zelinskie
c2d8aec157
Merge pull request #382 from caipre/patch-1
...
pgsql: Change layer name column data type
2017-05-19 20:07:53 -04:00
Nick Platt
aea74550e1
pgsql: Expand layer, namespace column widths
...
Presently the layer and namespace tables use type `varchar(128)` for
their respective name columns. For layer, this width works fine enough
using the sha256 digests provided by docker. However, if one wishes to
encode the image name into the layer name (eg, to avoid collisions like
in [0]), the limit of 128 bytes starts to feel a bit cramped. Bump to
256 bytes, since that "ought to be enough for anybody." (TM)
[0]: https://github.com/coreos/clair/issues/319
2017-05-19 18:22:04 -04:00
Jimmy Zelinskie
0305dde964
database/models: MetadataMap decodes from string
...
github.com/lib/pq began decoding text-like fields as strings to
Scanners.
See lib/pq@e2402a7cd1
2017-05-06 17:01:41 -04:00
Sida Chen
9306e99368
converted to structured logging by using logrus
...
changed from capnslog to logrus for logging JSON structured message.
finished issue #383
2017-05-04 13:59:57 -04:00
Tianon Gravi
b6ab5d8168
Add Ubuntu Zesty and Artful to UbuntuReleasesMapping
...
See also https://wiki.ubuntu.com/Releases
2017-04-26 14:45:32 -07:00
Jimmy Zelinskie
9e875f748d
database/pgsql: copy whole namespace
2017-02-23 02:05:33 -05:00
Jimmy Zelinskie
6a569fd945
move config to main / decentralize config
...
This puts config in its relevant location and moves functions around
loading config files into the main package.
As a side effect of removing cyclic imports for the API config, the
context library is no longer used.
2017-01-27 00:36:13 -05:00
Jimmy Zelinskie
9c63a63944
clair: mv updater clair and mv severity to db
2017-01-22 23:20:56 -05:00
Jimmy Zelinskie
3e4dc3834f
utils: remove string.go
2017-01-22 23:02:51 -05:00
Jimmy Zelinskie
e7f72ef5ad
utils: rm prometheus.go
2017-01-22 23:02:51 -05:00
Jimmy Zelinskie
343e24eb7e
clair: remove types
package
...
This removes the `types` package instead moving the contents to the
top-level clair package.
This change also renames the `Priority` type to `Severity` in order to
reduce confusion.
This change also removes the IsValid method and replaces it with a safe
constructor to avoid the creation of invalid values.
Many docstrings were tweaked in the making of this commit.
2017-01-22 23:02:51 -05:00
Jimmy Zelinskie
78cef02fda
pkg: cerrors -> commonerr
2017-01-22 23:02:50 -05:00
Jimmy Zelinskie
224ff82543
pgsql/migrations: fix dpkg default versionfmt
2017-01-20 14:42:33 -05:00
Jimmy Zelinskie
ca9f340a91
pgsql: only select distinct layers
2017-01-19 17:09:57 -05:00
Jimmy Zelinskie
ea73aa153d
pgsql: searchNotificationLayerIntroducingVulnerability order by layer ID
...
This fixes a bug where the API was returning Notification pages ordered
by LDFV.ID instead of by Layer ID.
2017-01-19 13:45:07 -05:00
Jimmy Zelinskie
627b98ef31
db/pgsql/feature: fix SQL error reporting
2017-01-03 17:06:08 -05:00
Jimmy Zelinskie
8d29bf860d
versionfmt: convert to using constant over literal
2017-01-03 16:00:20 -05:00
Jimmy Zelinskie
8df8170ba5
db/pgsql/migration: convert to pure SQL
2017-01-03 15:59:22 -05:00
Jimmy Zelinskie
9e39a26f26
backfill version_format column
2017-01-03 13:15:14 -05:00
Jimmy Zelinskie
033709eaea
add registerable version formats
...
Since we only ever used dpkg, this change shims everything into using
dpkg.
2016-12-30 12:51:24 -05:00
Jimmy Zelinskie
9338f28e82
psql/migrations: fix ordering
2016-12-25 19:25:57 -05:00
Jimmy Zelinskie
d4522e9c6e
api/v1: indexed layers for notifications
...
This change deprecates the old LayersIntroducingVulnerability for a new
one that orders output and contains an Index. This index is not
guaranteed to be consistent across multiple notifications, despite the
current Postgres implementation using the primary key of Layer table.
2016-12-06 19:23:33 -05:00
Quentin Machu
1fcae6abb8
Merge pull request #280 from coreos/add_idx_deleted_at
...
pgsql/migrations: add index on Vulnerability_Notification.deleted_at
2016-12-06 19:48:40 +01:00
Quentin Machu
7a3dd5c817
pgsql: Disable hashjoins to get introducing layers for notifications
2016-12-06 16:19:10 +01:00
Quentin Machu
eeb13a02ba
pgsql/migrations: add index on Vulnerability_Notification.deleted_at
...
`searchNotificationAvailable` never effectively use any indexes because:
- `notified_at < $1`, where $1 is a recent timestamp, returns the
majority of the table and therefore it is cheaper for PostgreSQL
to use a sequential scan on the table.
- there is no index for `deleted_at IS NULL`.
However, when Clair has been running for long enough, the grand majority
of rows (99%+) are expected to have a non-NULL `deleted_at` field. This
commit adds a new index on this very field in order to fetch the
remaining 1% in the blink of an eye.
In other words, instead of realizing a full table scan for each
`searchNotificationAvailable` query, we'll use the small branch of a new
index, reducing the total cost from over 30k to a mere 150 on a Clair
database that has already managed more than 1 000 000 notifications.
2016-12-06 14:39:52 +01:00
Jimmy Zelinskie
dab6e492b8
Merge pull request #279 from coreos/searchintro_optimize
...
pgsql: Reduce cost of GetNotification by 2.5x
2016-12-04 12:08:50 -05:00
Quentin Machu
dc8f71024f
pgsql: Reduce cost of GetNotification by 2.5
...
By delaying the Layer join to the very end, we can cut the query costs from 540,836 to 219,477.
See Pull Request for details.
2016-12-04 13:21:47 +01:00
Jimmy Zelinskie
7cff31a058
pgsql/migrations: add ldfv compound index
...
This speeds up the SearchNotificationLayerIntroducingVulnerability query
by an order magnitude.
2016-12-04 05:02:15 -05:00
Jimmy Zelinskie
9dc002621a
psql: add useful indexes
...
This adds some missing UNIQUE constraints and indexes for the
vulnerability table that should improve query performance.
2016-12-02 15:48:12 -05:00
Quentin Machu
7fddd68f4f
Merge pull request #263 from Quentin-M/rhel_unique_fixedin
...
pgsql: Do not insert entry in Vulnerability_FixedIn_Feature if existing
2016-11-13 13:25:08 +01:00
Quentin Machu
ec0aad9b7a
pgsql: Use booleans instead of varchar to return creation status
2016-11-12 15:42:59 +01:00
Quentin Machu
cd23262e41
pgsql: Do not insert entry in Vulnerability_FixedIn_Feature if existing
...
Fixes #238
2016-11-11 19:17:32 +01:00
Quentin Machu
b8865b2106
pgsql: Replace liamstask/goose by remind101/migrate
...
Fixes #93
2016-11-11 18:11:48 +01:00
Matt Moore
90cc8243ba
Add Ubuntu yakkety to the namespace mapping.
2016-10-24 21:08:02 -07:00
Quentin Machu
a03459d02e
Merge pull request #165 from Quentin-M/db_registration
...
Allow specifying datastore driver by config, relocate upgrade detection, mock datastore
2016-05-20 12:20:26 -05:00
Jimmy Zelinskie
5d8336acb3
pgsql: use subquery to plan GetNotification query ( #182 )
...
This change enables the query planner to wait and sort the result set of
our query rather than attempting to re-use the layer table's index for
the ORDER BY clause. Because the result set is always small, this makes
queries that were previous tens of seconds, now tens of milliseconds.
2016-05-20 13:10:00 -04:00
Quentin Machu
836d37b275
*: use path/filepath
instead of path
2016-05-20 12:01:31 -05:00
Quentin Machu
b99e2b50e2
database: Add some missing copyright headers
2016-05-20 12:01:31 -05:00
Quentin Machu
629d2ce662
database: Mock Datastore interface
2016-05-20 12:01:30 -05:00
Quentin Machu
a38fbf6cfe
worker/database: Move upgrade detection logic out of database to worker
2016-05-20 12:01:30 -05:00
Jimmy Zelinskie
51f9c5dcb4
pgsql: remove unnecessary join used in GetNotification ( #179 )
2016-05-19 11:45:38 -04:00
Quentin Machu
e7b960c05b
database: Allow specifying datastore driver by config
...
Fixes #145
2016-05-11 15:29:13 -07:00
Quentin Machu
79ba99bbea
database: Fix invalid error message
2016-03-18 12:25:01 -04:00
Jimmy Zelinskie
363cde29f4
psql: add debug message for duplicate layers
2016-03-08 14:39:53 -05:00
liangchenye
1a863a06cf
remove the useless pointer of NextPage field; check namespace notfound error
...
Signed-off-by: liangchenye <liangchenye@huawei.com>
2016-03-08 11:48:52 +08:00
liangchenye
48ffb2687a
use encrypt page in listVuln api
...
Signed-off-by: liangchenye <liangchenye@huawei.com>
2016-03-07 14:07:07 +08:00
liangchenye
a541e964e0
New API: list vulnerabilities by namespace
...
Signed-off-by: liangchenye <liangchenye@huawei.com>
2016-03-07 14:07:07 +08:00
Quentin Machu
9b191fb598
database: Find the FeatureVersion we try to insert before doing any lock
...
This commit is issued in order to limit the bottleneck that the
exclusive database lock on Vulnerability_Affects_FeautreVersion
introduces, when we inserting FeatureVersions. This slowdowns a bit
the FeatureVersion insertion on a mostly empty database but should
increase a lot the throughput and parallelism on a populated database.
2016-03-03 14:15:06 -05:00
Jimmy Zelinskie
500fc4e407
various: gofmt -s
2016-02-24 19:29:36 -05:00
Jimmy Zelinskie
8fd0aa162b
various: spelling corrections
2016-02-24 18:00:36 -05:00
Quentin Machu
84319507df
database: use constants to store queries
2016-02-24 16:40:40 -05:00
Quentin Machu
06531e01c5
database: disable hash/merge joins in FindLayer
...
Our experiments have shown that PostgreSQL 9.4 makes bad
planning decisions about:
- joining the layer tree to feature versions and feature
- joining the feature versions to affected/fixed feature version and vulnerabilities
It would for instance do a merge join between affected feature versions (300 rows, estimated
3000 rows) and fixed in feature version (100k rows). In this case, it is much more
preferred to use a nested loop.
2016-02-24 16:40:40 -05:00
Quentin Machu
18f2d7e672
database: modify join table in FindLayer to reduce cost by 3.5x
2016-02-24 16:40:40 -05:00
Quentin Machu
6d2eedf121
api/database: add the layer name that add each feature in getLayer
2016-02-24 16:39:25 -05:00
Quentin Machu
b5d8f9952e
database: fix notification test (wrong signature)
2016-02-24 16:36:45 -05:00
Quentin Machu
f0816d2c4d
database: add docs about the interface
2016-02-24 16:36:45 -05:00
Quentin Machu
d3b14106a9
database: ignore insertLayer collisions to make it truly idempotent
2016-02-24 16:36:45 -05:00
Quentin Machu
e3a25e5368
database: ignore min versions during new vulnerability insertions
2016-02-24 16:36:45 -05:00
Quentin Machu
7c11e4eb5d
updater/database: do not create notifications during the initial update
2016-02-24 16:36:45 -05:00
Quentin Machu
883be8769f
database: fix Ping() method in PostgreSQL's implementation
2016-02-24 16:36:45 -05:00
Quentin Machu
f8b4a52f8a
database: make notification tests more robust (old/new, update/delete vulnerabilities)
2016-02-24 16:36:45 -05:00
Quentin Machu
ccaaff000e
database: add created_at field for layers and vulnerabilities
2016-02-24 16:36:45 -05:00
Quentin Machu
94ece7bf2b
database: fix notification design and add vulnerability history
2016-02-24 16:36:45 -05:00
Jimmy Zelinskie
99f3552470
database: add Insert/DeleteVulnerabilityFix
2016-02-24 16:36:45 -05:00
Quentin Machu
03d904c620
database: improve PostgreSQL test inits and cleanups
2016-02-24 16:36:45 -05:00
Quentin Machu
5fdd9d1a07
*: add metadata support along with NVD CVSS
2016-02-24 16:36:45 -05:00
Quentin Machu
8f9779e232
database: cache feature version upon lookup
2016-02-24 16:34:54 -05:00
Quentin Machu
1e4ded6f2b
database: add ability to list namespaces
2016-02-24 16:34:54 -05:00
Quentin Machu
35df7ca0eb
database: fix feature version cache
2016-02-24 16:34:54 -05:00
Quentin Machu
8be18a0a01
database: write more of the notification system
2016-02-24 16:34:54 -05:00
Quentin Machu
3ecb8b69cb
updater: ignore "ubuntu-core" in the Ubuntu fetcher
2016-02-24 16:34:54 -05:00
Quentin Machu
d3d689a26a
database: don't prune locks when we renew one
2016-02-24 16:34:54 -05:00
Quentin Machu
2690800331
database: create notification during vulnerability insertion
2016-02-24 16:34:54 -05:00
Quentin Machu
baed60e19b
prometheus: add initial Prometheus support
2016-02-24 16:34:54 -05:00
Quentin Machu
ad0531acc7
notifier/database: refactor notification system and add initial Prometheus support
2016-02-24 16:34:54 -05:00
Quentin Machu
b8b7be3f81
*: remove health checker
2016-02-24 16:34:54 -05:00
Quentin Machu
63ebddfd36
database: add vulnerability deletion support
2016-02-24 16:34:54 -05:00
Quentin Machu
21f152c03e
database: fix keyvalue/notification tests
2016-02-24 16:34:54 -05:00
Quentin Machu
563b3825d8
database: let handleErrors deal with the not found case
2016-02-24 16:34:54 -05:00
Quentin Machu
c60d0054fa
notifier/database: draft new notification system
2016-02-24 16:34:54 -05:00
Quentin Machu
5759af5bcf
database: test and fix layer updates
2016-02-24 16:34:54 -05:00
Quentin Machu
726bd3c0c6
database/api: add layer deletion support
2016-02-24 16:34:54 -05:00
Quentin Machu
248fc7df72
database: fix cache collision (feature & feature versions)
2016-02-24 16:34:54 -05:00
Quentin Machu
82175dcfe9
*: add missing copyright headers
2016-02-24 16:34:54 -05:00
Quentin Machu
6e20993bac
api: simplify getLayer route and JSON output
2016-02-24 16:34:54 -05:00
Quentin Machu
92b734d0a4
database: remove an useless query in FindLayer
2016-02-24 16:34:54 -05:00
Quentin Machu
bd17dfb5e1
database: ensure that concurrent vulnerability/feature versions insertions work fine
2016-02-24 16:34:54 -05:00
Quentin Machu
74fc5b3e66
database: add missing transaction commits and close opened statement before inserting feature versions.
2016-02-24 16:34:54 -05:00
Quentin Machu
c5d1a8e5f7
database: update vulnerabilities only when necessary
2016-02-24 16:34:54 -05:00
Quentin Machu
77387af2ac
updater: port updater and its fetchers
2016-02-24 16:34:54 -05:00
Quentin Machu
1b53142e38
database: allow removing fixed packages in vulnerabilities
2016-02-24 16:32:21 -05:00
Quentin Machu
7c70fc1c20
database: add initial vulnerability support
2016-02-24 16:32:21 -05:00
Quentin Machu
3a786ae020
database: add lock support
2016-02-24 16:32:21 -05:00
Quentin Machu
6a9cf21fd4
database: log and mask SQL errors
2016-02-24 16:32:21 -05:00
Quentin Machu
970756cd5a
database: do insert/find layers (with their features and vulnerabilities)
2016-02-24 16:32:21 -05:00
Quentin Machu
2c150b015e
*: refactor & do initial work towards PostgreSQL implementation
2016-02-24 16:32:21 -05:00