Christian König
cc17fe18a9
Remove lines with ABP extended CSS selector
...
Co-authored-by: Adam Warner <me@adamwarner.co.uk>
Signed-off-by: Christian König <ckoenig@posteo.de>
1 year ago
Christian König
61ff5b2c76
Unifiy sed commands
...
Co-authored-by: RD WebDesign <github@rdwebdesign.com.br>
Signed-off-by: Christian König <ckoenig@posteo.de>
1 year ago
RD WebDesign
66bfa606a7
Using a better text for the comment
...
Signed-off-by: RD WebDesign <github@rdwebdesign.com.br>
1 year ago
RD WebDesign
32fb2e69ff
Spliting the regex into TLD_pattern and subdomain_pattern
...
Signed-off-by: RD WebDesign <github@rdwebdesign.com.br>
1 year ago
RD WebDesign
c71460e4b6
Allow TLD blocking using ABP style
...
This validates patterns without dots (only for abp style), allowing TLDs to be blocked
Signed-off-by: RD WebDesign <github@rdwebdesign.com.br>
1 year ago
Adam Warner
0656ceb149
Speed things up a bit with some humble greps. Consolodate regexes and remove the need for so many mapfile/arrays
...
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
Use temp files for parsing and remove when done.
Always rm the non-domains temp file.
exit 1 if gravity database creation fails.
Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>
Co-authored-by: Dan Schaper <dan.schaper@pi-hole.net>
Co-authored-by: DL6ER <dl6er@dl6er.de>
Co-authored-by: Adam Warner <me@adamwarner.co.uk>
1 year ago
kot0dama
76b7453f90
Add configurable GRAVITY_TMPDIR variable into setupVars
...
Signed-off-by: Loïc Gomez <89980752+kot0dama@users.noreply.github.com>
1 year ago
Christian König
71e262c37f
Revert "Allow final dot (root zone)"
...
This reverts commit 0b5da9f0da
.
Signed-off-by: Christian König <ckoenig@posteo.de>
1 year ago
Christian König
0b5da9f0da
Allow final dot (root zone)
...
Signed-off-by: Christian König <ckoenig@posteo.de>
1 year ago
Christian König
16385af3ef
Use dedicated pattern variable to make RegEx reusable
...
Signed-off-by: Christian König <ckoenig@posteo.de>
1 year ago
Christian König
6cb0be82ca
Add flag abp_domains into info table to signal if abp domains have been found
...
Signed-off-by: Christian König <ckoenig@posteo.de>
1 year ago
Christian König
821c7dc190
Add info when list cotains ABP style domains
...
Signed-off-by: Christian König <ckoenig@posteo.de>
1 year ago
Christian König
eedd93d782
Remove temporary files
...
Signed-off-by: Christian König <ckoenig@posteo.de>
1 year ago
Adam Warner
1f3f849106
Remove lines starting with ! or [ to account for ABP style comments and header
...
Also splits the piped "one-liner" in ParseFileIntoDomains into individually commented commands (makes for easier reading and debugging)
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
1 year ago
Christian König
ca74152d1d
Allow adding ABP style blocklists
...
Co-authored-by: Adam Warner <me@adamwarner.co.uk>
Signed-off-by: Christian König <ckoenig@posteo.de>
1 year ago
Christian König
a4bdf2454b
Don't use intermediate strings to filter false positives in gravity
...
Signed-off-by: Christian König <ckoenig@posteo.de>
1 year ago
Adam Warner
9939cf1d77
Rename some of the variables to hopefully make the process a little clearer
...
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
1 year ago
Adam Warner
79f4a7cef0
Maintain a list of valid hostnames (taken from the top of StevenBlack's host file), and discount them when calculating the number of "invalid" domains in a given list
...
Soften the output message when reporting on unsuable domains
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
1 year ago
Christian König
14ab586603
Start counting with 1
...
Signed-off-by: Christian König <ckoenig@posteo.de>
2 years ago
Christian König
372070ab39
Improve detection of incorrecr lines during gravity
...
Signed-off-by: Christian König <ckoenig@posteo.de>
2 years ago
Christian König
4004a93d1a
Don't double check for lines containing no domains in gravity
...
Signed-off-by: Christian König <ckoenig@posteo.de>
2 years ago
RD WebDesign
090727b1d3
Remove unnecessary return
...
Signed-off-by: RD WebDesign <github@rdwebdesign.com.br>
2 years ago
RD WebDesign
8b98fd06ed
Remove unused old code
...
Signed-off-by: RD WebDesign <github@rdwebdesign.com.br>
2 years ago
Christian König
230d6a435f
Semi hardcode the connection_timeout value
...
Signed-off-by: Christian König <ckoenig@posteo.de>
2 years ago
Christian König
53c0982c8b
Set connection timeout for curl
...
Signed-off-by: Christian König <ckoenig@posteo.de>
2 years ago
Adam Warner
64dc002c3f
Same per @rdwebdesign suggestion.
...
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2 years ago
Adam Warner
6c7a28a7b5
Dont use bash keywords/programs as variable names
2 years ago
Christian König
b486786041
Add database healthy checks to debug script
...
Fix health check output comparison in gravity script as well
Add note about waiting time
Signed-off-by: Christian König <ckoenig@posteo.de>
2 years ago
Adam Warner
0be987239e
Adds spellcheck action to the smoke-tests (also moves editorconfig-checker to smoketests away from it's own action)
...
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2 years ago
Peter Russell
052a1f294e
Update gravity.sh
...
Co-authored-by: yubiuser <ckoenig@posteo.de>
2 years ago
Peter Russell
325f20d5c2
Update gravity.sh
...
Co-authored-by: yubiuser <ckoenig@posteo.de>
2 years ago
Peter Russell
1c28da7bbd
Update gravity.sh
...
Co-authored-by: yubiuser <ckoenig@posteo.de>
2 years ago
jpgpi250
aba5e884eb
Update gravity.sh
...
Signed-off-by: jpgpi250 <jpgpi250@gmail.com>
2 years ago
DL6ER
8cbffa179d
Replace remaining sqlite3 calls by calls to our embedded pihole-FTL sqlite3 engine and remove sqlite3 as dependency in the installer.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2 years ago
DL6ER
534f9a63bf
Copy database tables earlier into the new gravity database to avoid foreign key contraint violations when adding gravity entries refering to an empty adlist table
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2 years ago
DL6ER
f0f5cc52d9
Use internal SQLite3 engine in more places in gravity.sh
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2 years ago
DL6ER
ce86157067
Fix gravity in case there are no adlists at all or all are disabled ( #4535 )
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2 years ago
DL6ER
3d3bb45a46
Merge pull request #4288 from pi-hole/new/gravity_repair
...
Implement fully-automated gravity database recovery method
2 years ago
DL6ER
d2a98ae954
Document -r recover force case
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2 years ago
DL6ER
2e1ce7fc87
Apply suggestions from code review
...
Co-authored-by: yubiuser <ckoenig@posteo.de>
2 years ago
DL6ER
54c58327f1
Merge pull request #4450 from pi-hole/unblock_NODATA
...
Unblock adlist domain during gravity run in NODATA mode
2 years ago
yubiuser
db5e94b14a
use +short and omit obsolet awk
...
Co-authored-by: DL6ER <DL6ER@users.noreply.github.com>
2 years ago
DL6ER
7167e6d5e4
Apply suggestions from code review
...
Co-authored-by: Dan Schaper <dan.schaper@pi-hole.net>
2 years ago
DL6ER
b06efb6ab7
Declare variables local
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2 years ago
DL6ER
ab4bce4787
Allow users to force recovery even when checks are okay using "pihole -g -r recover force"
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2 years ago
DL6ER
469c179b32
Return early from recovery routine when integrity checks didn't show any database errors.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2 years ago
DL6ER
190ab79606
Implement fully-automated gravity database recovery method.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
2 years ago
Dan Schaper
533a77d6d5
Add database function failure guards.
...
Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>
3 years ago
Dan Schaper
76ae75689c
Check for DNS before run.
...
Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>
3 years ago
Dan Schaper
a780fc59e2
Set DBFile permissions on creation.
...
Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>
3 years ago
Christian König
e243c562c2
Unblock adlist domain during gravity run in NODATA mode
...
Signed-off-by: Christian König <ckoenig@posteo.de>
3 years ago
Christian König
647ba6ec9d
Rename variables to improve comprehensibility
...
Signed-off-by: Christian König <ckoenig@posteo.de>
3 years ago
Christian König
bfda52ed79
Fix number of invalid domains
...
Co-authored-by: abesnier <besnier_antoine@yahoo.fr>
Signed-off-by: Christian König <ckoenig@posteo.de>
3 years ago
jbzdarkid
bc8150adfa
Clean up bash script formatting
...
Done with the help of beautysh (a python-based bash formatter)
Signed-off-by: jbzdarkid <jbzdarkid@gmail.com>
3 years ago
yubiuser
38bb4a4908
Remove unused wildcard_regex_converter.sh ( #4369 )
...
* Remove unused wildcard_regex_converter.sh
Signed-off-by: Christian König <ckoenig@posteo.de>
* Remove regexconverter
Signed-off-by: Christian König <ckoenig@posteo.de>
3 years ago
jpgpi250
3439045228
fix Pi-hole v5.4 update - syntax error #4322
...
Signed-off-by: jpgpi250 <jpgpi250@gmail.com>
3 years ago
Adam Warner
4ac4bc5c18
Merge branch 'development' into fix/gravity_no_locallist
3 years ago
jpgpi250
a2d5b8050a
Update gravity.sh
...
Signed-off-by: jpgpi250 <jpgpi250@gmail.com>
3 years ago
DL6ER
f90dffbf1a
Do not put hostname and pi.hole into local.list - this will be handled by FTL after FTL#1111 has been merged
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
3 years ago
DL6ER
f536718aaa
Use possibly custom gravity.db location in the copying SQL script
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
3 years ago
DL6ER
d25240fe9f
pihole -g should respose GRAVITYDB in pihole-FTL.conf
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
3 years ago
DL6ER
126b9ae381
Add missing three new columns to pihole -g -r
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
3 years ago
DL6ER
57fe3b6621
Merge pull request #3951 from pi-hole/new/gravity_adlist_infos
...
Store gravity details in gravity.db adlist table
3 years ago
Adam Warner
9ec012289f
Merge pull request #4017 from mwoolweaver/patch-2
...
fix suspected typo in comment
3 years ago
Michael Woolweaver
c42b97ddb2
fix typo in comment
...
Signed-off-by: Michael Woolweaver <m.woolweaver@icloud.com>
3 years ago
freddii
7c0c30fb0b
fixed typos
3 years ago
DL6ER
1791fe22f6
Merge pull request #3912 from mdujava/basic_auth
...
Fix validation of adlist url
3 years ago
DL6ER
a2625df5e2
Remove "The list is new" comment because it is superfluous
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
3 years ago
DL6ER
0944807491
Actually store correct number of domains for the individual lists (and not the sum of the so far collected number of domains)
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
3 years ago
DL6ER
a216848c1d
Only update time of last list change when we see a list for the first time or when it really changed content
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
3 years ago
DL6ER
8c56f54a1e
Compare checksum of downloaded list against older checksums to see if the list content changed since the last download
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
3 years ago
DL6ER
6b33b8b4c0
Store status of downloaded list (downloaded, using cache, some error, ...) and number of (in-)valid domains on this list in the gravity database. This updates the gravity databaes to version 14.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
3 years ago
Matej Dujava
ef0bdf6470
Fix validation of adlist url
...
Already existing regex validation will be used on url after removing @ (in case
its in separating userinfo and host).
Signed-off-by: Matej Dujava <mdujava@kocurkovo.cz>
Fixes: https://github.com/pi-hole/pi-hole/issues/3911
Fixes: 7d19ee1b: validate blocklist URL before adding to the database (#3237 )
4 years ago
user
922f795fe6
Adding comma to the adlist table format
...
To account for the Date Updated column in the adlist table adding a comma to the adlist to be imported into the database. This was causing intermittent failures on updates and therefore no domains would be blocked.
Signed-off-by: user <user@fedbrew.home>
Date: Sun Dec 6 06:03:30 2020 -0500
Committer: user <user@fedbrew.home>
On branch fix/gravity_date_updated
Your branch and 'origin/fix/gravity_date_updated' have diverged,
and have 1 and 1 different commits each, respectively.
(use "git pull" to merge the remote branch into yours)
Changes to be committed:
modified: gravity.sh
Signed-off-by: user <user@fedbrew.home>
4 years ago
Adam Warner
948337dd8e
Merge pull request #3740 from pi-hole/new/adlist_date_updated
...
Add date_updated field to adlist table
4 years ago
DL6ER
4fd412d7c3
Add date_updated field in adlist table set when a list changes.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
DL6ER
aa7c3b6852
Remove gravity optimization. Further investigations have shown that it is useless because the standard gravity run already produces an optimal database.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
Adam Warner
bb1a6243d6
Merge pull request #3669 from pi-hole/release/v5.1.2
...
Release/v5.1.2 to development
4 years ago
Christian König
27399a762a
Check for compression onyl once and print result
...
Signed-off-by: Christian König <ckoenig@posteo.de>
4 years ago
DL6ER
87da9084e6
Use compression (if available) when downloading the ad lists.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
DL6ER
527fc3c5cf
Also pidof -> pgrep in gravity.sh
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
Christian König
73963fecda
Use gravity's adlist_id in filename when saving downloaded adlist locally
...
Signed-off-by: Christian König <ckoenig@posteo.de>
4 years ago
Peter Lowe
dd205c55c1
Updated gravity.sh to allow semicolons
...
Signed-off-by: Peter Lowe <pgl@yoyo.org>
4 years ago
Adam Warner
9e96ff37de
Fix merge conflicts
...
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
4 years ago
DL6ER
288d487fc0
Allow ( and ) in adlist URLs.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
DL6ER
fa57c457f3
Boldify number of unique domains as this is the actually interesting number
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
Dan Schaper
0f20470a38
Merge pull request #3269 from pi-hole/tweak/hosts-comments
...
Add support for comments in HOSTS-like files
4 years ago
DL6ER
982c1b0059
Merge branch 'release/v5.0' into development
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
Adam Warner
7d19ee1b25
validate blocklist URL before adding to the database ( #3237 )
...
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
Co-authored-by: DL6ER <dl6er@dl6er.de>
4 years ago
DL6ER
7b15a88dc4
Strip comments from downloaded lists instead of discarding lines with comments altogether
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
XhmikosR
7b8611ced0
Assorted typo fixes. ( #3126 )
...
* Assorted typo fixes.
Signed-off-by: XhmikosR <xhmikosr@gmail.com>
Co-authored-by: Dan Schaper <dan.schaper@pi-hole.net>
4 years ago
DL6ER
3dd05606ca
Call it the received number of domains instead of the imported number as importing does only happen a bit later. Only show the number of invalid domains if there are invalid domains.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
DL6ER
1e8bfd33f5
Improve output
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
DL6ER
8131b5961c
Add comments to the code describing the changes.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
DL6ER
81d4531e10
Implement performant list checking routine.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
DL6ER
050e2963c7
Remove redundant code.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
DL6ER
3c09cd4a3a
Experimental output of matching line from shown warnings.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago
DL6ER
839fe32042
Fix issue with missing newline at the end of adlists ( #3144 )
...
* Also display non-fatal warnings during the database importing. Previously, we have only show warnings when there were also errors (errors are always fatal).
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Ensure there is always a newline on the last line.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Stickler linting
Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>
* Move sed command into subroutine to avoid code duplication.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Also unify comments.
Signed-off-by: DL6ER <dl6er@dl6er.de>
* Also unify comments.
Signed-off-by: DL6ER <dl6er@dl6er.de>
Co-authored-by: Dan Schaper <dan@glacialmagma.com>
4 years ago
DL6ER
d14ee26d6a
Merge pull request #3139 from pi-hole/fix/count_before_calling_FTL
...
Fix wrong number of blocking domains shown on the dashboard
4 years ago
DL6ER
52398052e9
Compute number of domains (and store it in the database) BEFORE calling FTL to re-read said value.
...
Signed-off-by: DL6ER <dl6er@dl6er.de>
4 years ago