Commit Graph

442 Commits

Author SHA1 Message Date
Jimmy Zelinskie
12c47e4066 docs: split http and json code blocks 2016-12-06 19:23:33 -05:00
Jimmy Zelinskie
d4522e9c6e api/v1: indexed layers for notifications
This change deprecates the old LayersIntroducingVulnerability for a new
one that orders output and contains an Index. This index is not
guaranteed to be consistent across multiple notifications, despite the
current Postgres implementation using the primary key of Layer table.
2016-12-06 19:23:33 -05:00
Quentin Machu
1fcae6abb8 Merge pull request #280 from coreos/add_idx_deleted_at
pgsql/migrations: add index on Vulnerability_Notification.deleted_at
2016-12-06 19:48:40 +01:00
Quentin Machu
83b5538c65 Merge pull request #281 from coreos/dis_hashjoins_introducing
pgsql: Disable hashjoins to get introducing layers for notifications
2016-12-06 19:48:33 +01:00
Quentin Machu
7a3dd5c817 pgsql: Disable hashjoins to get introducing layers for notifications 2016-12-06 16:19:10 +01:00
Quentin Machu
eeb13a02ba pgsql/migrations: add index on Vulnerability_Notification.deleted_at
`searchNotificationAvailable` never effectively use any indexes because:
- `notified_at < $1`, where $1 is a recent timestamp, returns the
  majority of the table and therefore it is cheaper for PostgreSQL
  to use a sequential scan on the table.
- there is no index for `deleted_at IS NULL`.
However, when Clair has been running for long enough, the grand majority
of rows (99%+) are expected to have a non-NULL `deleted_at` field. This
commit adds a new index on this very field in order to fetch the
remaining 1% in the blink of an eye.

In other words, instead of realizing a full table scan for each
`searchNotificationAvailable` query, we'll use the small branch of a new
index, reducing the total cost from over 30k to a mere 150 on a Clair
database that has already managed more than 1 000 000 notifications.
2016-12-06 14:39:52 +01:00
Quentin Machu
18e0018f80 Merge pull request #277 from jzelinskie/travispg
travis: add matrix for postgres
2016-12-04 19:00:52 +01:00
Quentin Machu
f5af78ed45 Merge branch 'master' into travispg 2016-12-04 19:00:47 +01:00
Jimmy Zelinskie
dab6e492b8 Merge pull request #279 from coreos/searchintro_optimize
pgsql: Reduce cost of GetNotification by 2.5x
2016-12-04 12:08:50 -05:00
Jimmy Zelinskie
2fe4a464e1 Merge pull request #278 from jzelinskie/layerdiffindex
pgsql/migrations: add ldfv compound index
2016-12-04 12:06:27 -05:00
Quentin Machu
dc8f71024f pgsql: Reduce cost of GetNotification by 2.5
By delaying the Layer join to the very end, we can cut the query costs from 540,836 to 219,477.

See Pull Request for details.
2016-12-04 13:21:47 +01:00
Jimmy Zelinskie
7cff31a058 pgsql/migrations: add ldfv compound index
This speeds up the SearchNotificationLayerIntroducingVulnerability query
by an order magnitude.
2016-12-04 05:02:15 -05:00
Jimmy Zelinskie
4fab327397 travis: add matrix for postgres 2016-12-03 21:00:21 -05:00
Jimmy Zelinskie
026f64aa82 Merge pull request #276 from jzelinskie/index
psql: add useful indexes
2016-12-02 16:07:48 -05:00
Jimmy Zelinskie
9dc002621a psql: add useful indexes
This adds some missing UNIQUE constraints and indexes for the
vulnerability table that should improve query performance.
2016-12-02 15:48:12 -05:00
Quentin Machu
39854ac64d Merge pull request #274 from JensPiegsa/patch-1
Restart clair until postgres is up
2016-11-22 13:26:11 +01:00
Jens Piegsa
9d75a90a8a Restart clair until postgres is up 2016-11-22 12:52:25 +01:00
Quentin Machu
639d305cf8 Merge pull request #271 from Quentin-M/nvd_severity
updater: Set vulns' Severity from NVD metadata fetcher if unknown
2016-11-18 21:31:47 +01:00
Quentin Machu
1faf27ba18 utils: Fix OVAL's log statements 2016-11-18 19:00:13 +01:00
Quentin Machu
96398465de updater: Set vulns' Severity from NVD metadata fetcher if unknown
If a Vulnerability that goes through the NVD metadata fetcher has an
empty or Unknown Severity, then use the CVSS score to set one. This
will help to get a more consistent database when a vulnerability source
does not provide this information.
2016-11-18 19:00:13 +01:00
Quentin Machu
1c3daa23b9 updater: minimize vulns' lock duration in the NVD metadata fetcher 2016-11-18 18:09:59 +01:00
Quentin Machu
e38b67a513 Merge pull request #270 from Quentin-M/imp_docs
docs: improve GET/POST /v1/layers documentation
2016-11-18 01:03:16 +01:00
Quentin Machu
37a58260db docs: improve GET/POST /v1/layers documentation 2016-11-17 17:54:23 +01:00
Quentin Machu
7fddd68f4f Merge pull request #263 from Quentin-M/rhel_unique_fixedin
pgsql: Do not insert entry in Vulnerability_FixedIn_Feature if existing
2016-11-13 13:25:08 +01:00
Quentin Machu
ec0aad9b7a pgsql: Use booleans instead of varchar to return creation status 2016-11-12 15:42:59 +01:00
Quentin Machu
43cbc04356 Merge pull request #261 from Quentin-M/replace_goose
Replace liamstask/goose by remind101/migrate
2016-11-11 19:26:04 +01:00
Quentin Machu
cd23262e41 pgsql: Do not insert entry in Vulnerability_FixedIn_Feature if existing
Fixes #238
2016-11-11 19:17:32 +01:00
Jimmy Zelinskie
eed4a9a7dd Merge pull request #262 from jzelinskie/travis
travis: update to use Go 1.7, glide
2016-11-11 12:23:32 -05:00
Quentin Machu
b8865b2106 pgsql: Replace liamstask/goose by remind101/migrate
Fixes #93
2016-11-11 18:11:48 +01:00
Jimmy Zelinskie
2d0be7ccf4 travis: update to use Go 1.7, glide 2016-11-11 12:07:59 -05:00
Quentin Machu
43969afa60 Merge pull request #257 from mattmoor/yakkety
Add Ubuntu yakkety to the namespace mapping.
2016-10-25 09:50:18 +02:00
Matt Moore
90cc8243ba Add Ubuntu yakkety to the namespace mapping. 2016-10-24 21:08:02 -07:00
Jimmy Zelinskie
861cba0f49 README: s/1.2.2/1.2.4 2016-10-24 12:42:22 -04:00
Jimmy Zelinskie
97347ec44d Merge pull request #199 from openSUSE/feature/opensuse
Add openSUSE/SLES support
2016-09-23 10:19:24 -04:00
Jordi Massaguer Pla
b8ceb0c461 Integrated a fetcher for openSUSE and for SUSE Linux Enterprise
We extracted oval parser from rhel and used that for opensuse and
SUSE Linux Enterpise

Signed-off-by: Thomas Boerger <tboerger@suse.de>
Signed-off-by: Jordi Massaguer Pla <jmassaguerpla@suse.de>
2016-09-23 12:00:00 +02:00
Jimmy Zelinskie
051564facd Merge pull request #236 from robszumski/doc-link
readme: add link to docs dir
2016-09-06 15:08:26 -04:00
Jimmy Zelinskie
eb6117c372 Merge pull request #235 from jzelinskie/doc-move
docs: move to standard Documentation dir
2016-09-06 15:08:08 -04:00
Jimmy Zelinskie
80870bff84 Merge pull request #229 from vbatts/redhatrelease_detector
Redhatrelease detector
2016-09-06 15:04:17 -04:00
Steven Trescinski
28fbfc72e7 Fixed broken quay.io links (#232) 2016-08-16 14:22:22 -04:00
Vincent Batts
ce8d31bbb3 redhatrelease: override match for RHEL hosts
Until https://github.com/coreos/clair/pull/193 is merged, having
vulnerabilities that are tagged both rhel and centos would duplicate in
the database or use a change that requires a migration.

But presently due to the fetcher logic, the rhel provided
vulnerabilities are labelled for centos, and then the namespace does not
match and therefore not tested against.

So until such a day that a vulnerability could have both rhel and centos
label, then hack this in. It'll accomplish the same during this interim.

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-08-12 15:35:32 -04:00
Vincent Batts
d88f797821 osrelease-detector: avoid colliding with other detectors
Due to the detector registration and fact that their in a non-ordered
map, it is random whether the osrelease or redhatrelease detector would
hit. And likely resulted in alternately formatted namespace strings.

This change causes the osrelease to not detect when data has
centos-release or redhat-release, which is not _great_ because if the
redhatrelease detector is not compiled in, then that would not be a
fallback that the osrelease detector could rely on. :-\

Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-08-12 15:32:12 -04:00
Vincent Batts
c28d2b3a66 namespace: add debug output
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
2016-08-12 15:31:44 -04:00
Jimmy Zelinskie
4bc6416132 README: include data licenses for data sources (#219) 2016-07-21 10:31:46 -04:00
Quentin Machu
be97db5261 updater: enable fetching of RHEL 5 vulnerabilities (#217)
The RHEL updater currently ignores vulnerabilities for CentOS <= 5.
s the naming of the constant firstConsideredRHEL suggests it, it
should actually considers CentOS 5 and ignores CentOS < 5.

Fixes #215
2016-07-15 11:54:47 -04:00
Quentin Machu
c1a7144113 Merge pull request #216 from optiopay/doc-klar-ref
README: add reference to Klar tool
2016-07-15 13:07:44 +02:00
Alexey Miroshkin
c4281b3a3c README: add reference to Klar tool
This update adds reference to Klar tool. Klar is a simple integration
tool designed to be used in scripts and CI.
2016-07-15 08:59:29 +02:00
Jimmy Zelinskie
b45b625fc8 improve v1 api docs header legibility
GitHub has changed fonts and as a result h6s are extremely small.
2016-07-11 14:35:35 -04:00
Quentin Machu
09d397eca9 Merge pull request #205 from Quentin-M/readme_v122
Replace latest by v1.2.2 in Quick Start & Add various talks
2016-06-15 23:59:32 +02:00
Quentin Machu
0733cd9077 Merge pull request #206 from Quentin-M/godeps_implsubmod
godeps: Remove implicit git submodules
2016-06-15 20:54:31 +02:00
Quentin Machu
213468a6d5 godeps: Remove implicit git submodules 2016-06-15 00:15:59 +02:00