arno/clair
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
785 Commits 5 Branches 24 Tags 158 MiB
0c2e5e73c2
Commit Graph

139 Commits

Author SHA1 Message Date
Quentin Machu
1fcae6abb8 Merge pull request #280 from coreos/add_idx_deleted_at 
pgsql/migrations: add index on Vulnerability_Notification.deleted_at
 2016-12-06 19:48:40 +01:00
Quentin Machu
7a3dd5c817 pgsql: Disable hashjoins to get introducing layers for notifications 2016-12-06 16:19:10 +01:00
Quentin Machu
eeb13a02ba pgsql/migrations: add index on Vulnerability_Notification.deleted_at 
`searchNotificationAvailable` never effectively use any indexes because:
- `notified_at < $1`, where $1 is a recent timestamp, returns the
  majority of the table and therefore it is cheaper for PostgreSQL
  to use a sequential scan on the table.
- there is no index for `deleted_at IS NULL`.
However, when Clair has been running for long enough, the grand majority
of rows (99%+) are expected to have a non-NULL `deleted_at` field. This
commit adds a new index on this very field in order to fetch the
remaining 1% in the blink of an eye.

In other words, instead of realizing a full table scan for each
`searchNotificationAvailable` query, we'll use the small branch of a new
index, reducing the total cost from over 30k to a mere 150 on a Clair
database that has already managed more than 1 000 000 notifications.
 2016-12-06 14:39:52 +01:00
Jimmy Zelinskie
dab6e492b8 Merge pull request #279 from coreos/searchintro_optimize 
pgsql: Reduce cost of GetNotification by 2.5x
 2016-12-04 12:08:50 -05:00
Quentin Machu
dc8f71024f pgsql: Reduce cost of GetNotification by 2.5 
By delaying the Layer join to the very end, we can cut the query costs from 540,836 to 219,477.

See Pull Request for details.
 2016-12-04 13:21:47 +01:00
Jimmy Zelinskie
7cff31a058 pgsql/migrations: add ldfv compound index 
This speeds up the SearchNotificationLayerIntroducingVulnerability query
by an order magnitude.
 2016-12-04 05:02:15 -05:00
Jimmy Zelinskie
9dc002621a psql: add useful indexes 
This adds some missing UNIQUE constraints and indexes for the
vulnerability table that should improve query performance.
 2016-12-02 15:48:12 -05:00
Quentin Machu
7fddd68f4f Merge pull request #263 from Quentin-M/rhel_unique_fixedin 
pgsql: Do not insert entry in Vulnerability_FixedIn_Feature if existing
 2016-11-13 13:25:08 +01:00
Quentin Machu
ec0aad9b7a pgsql: Use booleans instead of varchar to return creation status 2016-11-12 15:42:59 +01:00
Quentin Machu
cd23262e41 pgsql: Do not insert entry in Vulnerability_FixedIn_Feature if existing 
Fixes #238
 2016-11-11 19:17:32 +01:00
Quentin Machu
b8865b2106 pgsql: Replace liamstask/goose by remind101/migrate 
Fixes #93
 2016-11-11 18:11:48 +01:00
Matt Moore
90cc8243ba Add Ubuntu yakkety to the namespace mapping. 2016-10-24 21:08:02 -07:00
Quentin Machu
a03459d02e Merge pull request #165 from Quentin-M/db_registration 
Allow specifying datastore driver by config, relocate upgrade detection, mock datastore
 2016-05-20 12:20:26 -05:00
Jimmy Zelinskie
5d8336acb3 pgsql: use subquery to plan GetNotification query (#182) 
This change enables the query planner to wait and sort the result set of
our query rather than attempting to re-use the layer table's index for
the ORDER BY clause. Because the result set is always small, this makes
queries that were previous tens of seconds, now tens of milliseconds.
 2016-05-20 13:10:00 -04:00
Quentin Machu
836d37b275 *: use path/filepath instead of path 2016-05-20 12:01:31 -05:00
Quentin Machu
b99e2b50e2 database: Add some missing copyright headers 2016-05-20 12:01:31 -05:00
Quentin Machu
629d2ce662 database: Mock Datastore interface 2016-05-20 12:01:30 -05:00
Quentin Machu
a38fbf6cfe worker/database: Move upgrade detection logic out of database to worker 2016-05-20 12:01:30 -05:00
Jimmy Zelinskie
51f9c5dcb4 pgsql: remove unnecessary join used in GetNotification (#179) 2016-05-19 11:45:38 -04:00
Quentin Machu
e7b960c05b database: Allow specifying datastore driver by config 
Fixes #145
 2016-05-11 15:29:13 -07:00
Quentin Machu
79ba99bbea database: Fix invalid error message 2016-03-18 12:25:01 -04:00
Jimmy Zelinskie
363cde29f4 psql: add debug message for duplicate layers 2016-03-08 14:39:53 -05:00
liangchenye
1a863a06cf remove the useless pointer of NextPage field; check namespace notfound error 
Signed-off-by: liangchenye <liangchenye@huawei.com>
 2016-03-08 11:48:52 +08:00
liangchenye
48ffb2687a use encrypt page in listVuln api 
Signed-off-by: liangchenye <liangchenye@huawei.com>
 2016-03-07 14:07:07 +08:00
liangchenye
a541e964e0 New API: list vulnerabilities by namespace 
Signed-off-by: liangchenye <liangchenye@huawei.com>
 2016-03-07 14:07:07 +08:00
Quentin Machu
9b191fb598 database: Find the FeatureVersion we try to insert before doing any lock 
This commit is issued in order to limit the bottleneck that the
exclusive database lock on Vulnerability_Affects_FeautreVersion
introduces, when we inserting FeatureVersions. This slowdowns a bit
the FeatureVersion insertion on a mostly empty database but should
increase a lot the throughput and parallelism on a populated database.
 2016-03-03 14:15:06 -05:00
Jimmy Zelinskie
500fc4e407 various: gofmt -s 2016-02-24 19:29:36 -05:00
Jimmy Zelinskie
8fd0aa162b various: spelling corrections 2016-02-24 18:00:36 -05:00
Quentin Machu
84319507df database: use constants to store queries 2016-02-24 16:40:40 -05:00
Quentin Machu
06531e01c5 database: disable hash/merge joins in FindLayer 
Our experiments have shown that PostgreSQL 9.4 makes bad
planning decisions about:
- joining the layer tree to feature versions and feature
- joining the feature versions to affected/fixed feature version and vulnerabilities
It would for instance do a merge join between affected feature versions (300 rows, estimated
3000 rows) and fixed in feature version (100k rows). In this case, it is much more
preferred to use a nested loop.
 2016-02-24 16:40:40 -05:00
Quentin Machu
18f2d7e672 database: modify join table in FindLayer to reduce cost by 3.5x 2016-02-24 16:40:40 -05:00
Quentin Machu
6d2eedf121 api/database: add the layer name that add each feature in getLayer 2016-02-24 16:39:25 -05:00
Quentin Machu
b5d8f9952e database: fix notification test (wrong signature) 2016-02-24 16:36:45 -05:00
Quentin Machu
f0816d2c4d database: add docs about the interface 2016-02-24 16:36:45 -05:00
Quentin Machu
d3b14106a9 database: ignore insertLayer collisions to make it truly idempotent 2016-02-24 16:36:45 -05:00
Quentin Machu
e3a25e5368 database: ignore min versions during new vulnerability insertions 2016-02-24 16:36:45 -05:00
Quentin Machu
7c11e4eb5d updater/database: do not create notifications during the initial update 2016-02-24 16:36:45 -05:00
Quentin Machu
883be8769f database: fix Ping() method in PostgreSQL's implementation 2016-02-24 16:36:45 -05:00
Quentin Machu
f8b4a52f8a database: make notification tests more robust (old/new, update/delete vulnerabilities) 2016-02-24 16:36:45 -05:00
Quentin Machu
ccaaff000e database: add created_at field for layers and vulnerabilities 2016-02-24 16:36:45 -05:00
Quentin Machu
94ece7bf2b database: fix notification design and add vulnerability history 2016-02-24 16:36:45 -05:00
Jimmy Zelinskie
99f3552470 database: add Insert/DeleteVulnerabilityFix 2016-02-24 16:36:45 -05:00
Quentin Machu
03d904c620 database: improve PostgreSQL test inits and cleanups 2016-02-24 16:36:45 -05:00
Quentin Machu
5fdd9d1a07 *: add metadata support along with NVD CVSS 2016-02-24 16:36:45 -05:00
Quentin Machu
8f9779e232 database: cache feature version upon lookup 2016-02-24 16:34:54 -05:00
Quentin Machu
1e4ded6f2b database: add ability to list namespaces 2016-02-24 16:34:54 -05:00
Quentin Machu
35df7ca0eb database: fix feature version cache 2016-02-24 16:34:54 -05:00
Quentin Machu
8be18a0a01 database: write more of the notification system 2016-02-24 16:34:54 -05:00
Quentin Machu
3ecb8b69cb updater: ignore "ubuntu-core" in the Ubuntu fetcher 2016-02-24 16:34:54 -05:00
Quentin Machu
d3d689a26a database: don't prune locks when we renew one 2016-02-24 16:34:54 -05:00
Quentin Machu
2690800331 database: create notification during vulnerability insertion 2016-02-24 16:34:54 -05:00
Quentin Machu
baed60e19b prometheus: add initial Prometheus support 2016-02-24 16:34:54 -05:00
Quentin Machu
ad0531acc7 notifier/database: refactor notification system and add initial Prometheus support 2016-02-24 16:34:54 -05:00
Quentin Machu
b8b7be3f81 *: remove health checker 2016-02-24 16:34:54 -05:00
Quentin Machu
63ebddfd36 database: add vulnerability deletion support 2016-02-24 16:34:54 -05:00
Quentin Machu
21f152c03e database: fix keyvalue/notification tests 2016-02-24 16:34:54 -05:00
Quentin Machu
563b3825d8 database: let handleErrors deal with the not found case 2016-02-24 16:34:54 -05:00
Quentin Machu
c60d0054fa notifier/database: draft new notification system 2016-02-24 16:34:54 -05:00
Quentin Machu
5759af5bcf database: test and fix layer updates 2016-02-24 16:34:54 -05:00
Quentin Machu
726bd3c0c6 database/api: add layer deletion support 2016-02-24 16:34:54 -05:00
Quentin Machu
248fc7df72 database: fix cache collision (feature & feature versions) 2016-02-24 16:34:54 -05:00
Quentin Machu
82175dcfe9 *: add missing copyright headers 2016-02-24 16:34:54 -05:00
Quentin Machu
6e20993bac api: simplify getLayer route and JSON output 2016-02-24 16:34:54 -05:00
Quentin Machu
92b734d0a4 database: remove an useless query in FindLayer 2016-02-24 16:34:54 -05:00
Quentin Machu
bd17dfb5e1 database: ensure that concurrent vulnerability/feature versions insertions work fine 2016-02-24 16:34:54 -05:00
Quentin Machu
74fc5b3e66 database: add missing transaction commits and close opened statement before inserting feature versions. 2016-02-24 16:34:54 -05:00
Quentin Machu
c5d1a8e5f7 database: update vulnerabilities only when necessary 2016-02-24 16:34:54 -05:00
Quentin Machu
77387af2ac updater: port updater and its fetchers 2016-02-24 16:34:54 -05:00
Quentin Machu
1b53142e38 database: allow removing fixed packages in vulnerabilities 2016-02-24 16:32:21 -05:00
Quentin Machu
7c70fc1c20 database: add initial vulnerability support 2016-02-24 16:32:21 -05:00
Quentin Machu
3a786ae020 database: add lock support 2016-02-24 16:32:21 -05:00
Quentin Machu
6a9cf21fd4 database: log and mask SQL errors 2016-02-24 16:32:21 -05:00
Quentin Machu
970756cd5a database: do insert/find layers (with their features and vulnerabilities) 2016-02-24 16:32:21 -05:00
Quentin Machu
2c150b015e *: refactor & do initial work towards PostgreSQL implementation 2016-02-24 16:32:21 -05:00
Quentin Machu
712aa11b8b updater: Add support for Ubuntu Vivid Core and ignore Vivid PhoneOverlay 
Reacts to https://bazaar.launchpad.net/~ubuntu-security/ubuntu-cve-tracker/master/revision/10488
 2016-01-25 13:04:39 -05:00
Quentin Machu
32747a5f25 database: Don't ignore empty results in toValue(s)() 
There is apparently no reason to ignore empty results - it was probably the case in the past (`null` value).

["", "v"] should be considered invalid by toValue() because it represents two values.
["", "v"] should be returned as it by toValues(), not trimming "".

Tests passes, it will hopefully not cause any issue in prod.
 2015-12-15 15:20:38 -05:00
Quentin Machu
eb7e5d5c74 main: Use configuration file instead of flags and simplify app extension. 
Clair will now use a YAML configuration file instead of command line
arguments as the number of parameters grows.

Also, Clair now exposes a Boot() func that allows everyone to easily
create their own project and load dynamically their own fetchers/updaters.
 2015-12-08 11:50:52 -05:00
Quentin Machu
3fe3f3a4c7 database: Update cayley and use Triple instead of Quad 2015-12-06 20:15:40 -05:00
Quentin Machu
9fc29e291c database: put missing predicates in consts and un-expose some of them 
Fixes #16
 2015-12-04 16:50:18 -05:00
Quentin Machu
46fffdfc81 Merge pull request #33 from Quentin-M/insertvulns 
database: Improve InsertVulnerabilities.
 2015-12-04 15:49:56 -05:00
Quentin Machu
8285c567c8 database: Improve InsertVulnerabilities. 2015-12-04 14:42:58 -05:00
Quentin Machu
e444e93c97 api/database: Add the ability to delete layers 2015-12-04 14:42:21 -05:00
Quentin Machu
9db0e63401 api: Specify what packages cause the layer to have vulnerabilities. 2015-12-01 17:02:49 -05:00
Quentin Machu
cfa960d619 database: Update Cayley to fix slow deletions 
subject, predicate, object and labels are not indexed, thus, using a where clause on these fields for the DELETE statement does a full-table scan. Using *_hash columns instead will use the indexes.
 2015-11-18 18:53:00 -05:00
Quentin Machu
3a1d0602fb database: Use an estimator in Cayley's Size() w/ PostgreSQL 2015-11-16 16:22:16 -05:00
Quentin Machu
f229083e1e database/worker: Remove useless log message 2015-11-16 13:21:06 -05:00
Quentin Machu
b0142e1982 database: reduce pruneLocks/Unlock transaction. 
pruneLocks could create deadlocked transactions on PostgreSQL if multiple locks expired and pruneLocks is called by multiple instances. Also adds some logging.
 2015-11-16 12:06:42 -05:00
Quentin Machu
7f1ff8f979 database: reduce InsertPackages transaction 
Inserting packages in a single transaction does not actually buy us anything as we often delete quads during an insertion and thus, Cayley could not use COPY and do a single round-trip. Inserting multiple packages in a single transaction actually creates deadlocks when a transaction tries to insert (A,B) and another one tries to insert (B,A).
 2015-11-13 18:06:01 -05:00
Quentin Machu
3ec262dd51 Initial commit 2015-11-13 14:11:28 -05:00