arno/pi-hole
1
0
Fork 0
mirror of https://github.com/pi-hole/pi-hole synced 2024-11-19 14:48:13 +00:00
Code Issues Releases Wiki Activity
6,144 Commits 8 Branches 109 Tags 19 MiB
82a83c497d
Commit Graph

531 Commits

Author SHA1 Message Date
Adam Warner
1f3f849106
Remove lines starting with ! or [ to account for ABP style comments and header 
Also splits the piped "one-liner" in ParseFileIntoDomains into individually commented commands (makes for easier reading and debugging)

Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2023-02-22 21:21:38 +01:00
Christian König
ca74152d1d
Allow adding ABP style blocklists 
Co-authored-by: Adam Warner <me@adamwarner.co.uk>
Signed-off-by: Christian König <ckoenig@posteo.de>
 2023-02-22 21:21:13 +01:00
Christian König
a4bdf2454b
Don't use intermediate strings to filter false positives in gravity 
Signed-off-by: Christian König <ckoenig@posteo.de>
 2023-02-16 10:39:21 +01:00
Adam Warner
9939cf1d77
Rename some of the variables to hopefully make the process a little clearer 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2023-01-21 23:57:49 +00:00
Adam Warner
79f4a7cef0
Maintain a list of valid hostnames (taken from the top of StevenBlack's host file), and discount them when calculating the number of "invalid" domains in a given list 
Soften the output message when reporting on unsuable domains

Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2023-01-20 18:52:14 +00:00
Christian König
14ab586603
Start counting with 1 
Signed-off-by: Christian König <ckoenig@posteo.de>
 2022-12-11 15:12:28 +01:00
Christian König
372070ab39
Improve detection of incorrecr lines during gravity 
Signed-off-by: Christian König <ckoenig@posteo.de>
 2022-12-11 15:12:28 +01:00
Christian König
4004a93d1a
Don't double check for lines containing no domains in gravity 
Signed-off-by: Christian König <ckoenig@posteo.de>
 2022-12-11 15:12:28 +01:00
RD WebDesign
090727b1d3
Remove unnecessary return 
Signed-off-by: RD WebDesign <github@rdwebdesign.com.br>
 2022-10-17 15:24:49 -03:00
RD WebDesign
8b98fd06ed
Remove unused old code 
Signed-off-by: RD WebDesign <github@rdwebdesign.com.br>
 2022-10-16 15:54:24 -03:00
Christian König
230d6a435f
Semi hardcode the connection_timeout value 
Signed-off-by: Christian König <ckoenig@posteo.de>
 2022-08-31 21:41:57 +02:00
Christian König
53c0982c8b
Set connection timeout for curl 
Signed-off-by: Christian König <ckoenig@posteo.de>
 2022-08-31 14:43:20 +02:00
Adam Warner
64dc002c3f
Same per @rdwebdesign suggestion. 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2022-08-01 20:38:31 +01:00
Adam Warner
6c7a28a7b5
Dont use bash keywords/programs as variable names 2022-08-01 18:38:15 +01:00
Christian König
b486786041
Add database healthy checks to debug script 
Fix health check output comparison in gravity script as well
Add note about waiting time

Signed-off-by: Christian König <ckoenig@posteo.de>
 2022-07-10 23:32:47 +02:00
Adam Warner
0be987239e
Adds spellcheck action to the smoke-tests (also moves editorconfig-checker to smoketests away from it's own action) 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2022-07-02 14:04:08 +01:00
Peter Russell
052a1f294e
Update gravity.sh 
Co-authored-by: yubiuser <ckoenig@posteo.de>
 2022-04-20 10:44:05 +02:00
Peter Russell
325f20d5c2
Update gravity.sh 
Co-authored-by: yubiuser <ckoenig@posteo.de>
 2022-04-20 10:43:56 +02:00
Peter Russell
1c28da7bbd
Update gravity.sh 
Co-authored-by: yubiuser <ckoenig@posteo.de>
 2022-04-20 10:43:39 +02:00
jpgpi250
aba5e884eb Update gravity.sh 
Signed-off-by: jpgpi250 <jpgpi250@gmail.com>
 2022-04-20 10:19:44 +02:00
DL6ER
8cbffa179d
Replace remaining sqlite3 calls by calls to our embedded pihole-FTL sqlite3 engine and remove sqlite3 as dependency in the installer. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2022-01-30 11:18:17 +01:00
DL6ER
534f9a63bf
Copy database tables earlier into the new gravity database to avoid foreign key contraint violations when adding gravity entries refering to an empty adlist table 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2022-01-30 10:36:20 +01:00
DL6ER
f0f5cc52d9
Use internal SQLite3 engine in more places in gravity.sh 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2022-01-29 22:39:45 +01:00
DL6ER
ce86157067
Fix gravity in case there are no adlists at all or all are disabled (#4535) 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2022-01-08 14:15:26 +01:00
DL6ER
3d3bb45a46
Merge pull request #4288 from pi-hole/new/gravity_repair 
Implement fully-automated gravity database recovery method
 2021-12-22 21:08:01 +01:00
DL6ER
d2a98ae954
Document -r recover force case 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2021-12-22 19:53:52 +01:00
DL6ER
2e1ce7fc87
Apply suggestions from code review 
Co-authored-by: yubiuser <ckoenig@posteo.de>
 2021-12-22 19:52:08 +01:00
DL6ER
54c58327f1
Merge pull request #4450 from pi-hole/unblock_NODATA 
Unblock adlist domain during gravity run in NODATA mode
 2021-12-21 22:08:14 +01:00
yubiuser
db5e94b14a
use +short and omit obsolet awk 
Co-authored-by: DL6ER <DL6ER@users.noreply.github.com>
 2021-12-21 22:01:34 +01:00
DL6ER
7167e6d5e4
Apply suggestions from code review 
Co-authored-by: Dan Schaper <dan.schaper@pi-hole.net>
 2021-12-21 16:20:02 +01:00
DL6ER
b06efb6ab7
Declare variables local 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2021-12-21 14:00:46 +01:00
DL6ER
ab4bce4787
Allow users to force recovery even when checks are okay using "pihole -g -r recover force" 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2021-12-21 13:57:03 +01:00
DL6ER
469c179b32
Return early from recovery routine when integrity checks didn't show any database errors. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2021-12-21 13:57:03 +01:00
DL6ER
190ab79606
Implement fully-automated gravity database recovery method. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2021-12-21 13:57:03 +01:00
Dan Schaper
533a77d6d5
Add database function failure guards. 
Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>
 2021-12-20 11:36:55 -08:00
Dan Schaper
76ae75689c
Check for DNS before run. 
Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>
 2021-12-20 11:09:11 -08:00
Dan Schaper
a780fc59e2
Set DBFile permissions on creation. 
Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>
 2021-12-20 10:56:42 -08:00
Christian König
e243c562c2
Unblock adlist domain during gravity run in NODATA mode 
Signed-off-by: Christian König <ckoenig@posteo.de>
 2021-12-03 09:17:19 +01:00
Christian König
647ba6ec9d
Rename variables to improve comprehensibility 
Signed-off-by: Christian König <ckoenig@posteo.de>
 2021-12-02 23:13:01 +01:00
Christian König
bfda52ed79
Fix number of invalid domains 
Co-authored-by: abesnier <besnier_antoine@yahoo.fr>
Signed-off-by: Christian König <ckoenig@posteo.de>
 2021-12-01 11:07:17 +01:00
jbzdarkid
bc8150adfa Clean up bash script formatting 
Done with the help of beautysh (a python-based bash formatter)

Signed-off-by: jbzdarkid <jbzdarkid@gmail.com>
 2021-11-25 14:12:09 -08:00
yubiuser
38bb4a4908
Remove unused wildcard_regex_converter.sh (#4369) 
* Remove unused wildcard_regex_converter.sh

Signed-off-by: Christian König <ckoenig@posteo.de>

* Remove regexconverter

Signed-off-by: Christian König <ckoenig@posteo.de>
 2021-10-05 13:09:16 +01:00
jpgpi250
3439045228 fix Pi-hole v5.4 update - syntax error #4322 
Signed-off-by: jpgpi250 <jpgpi250@gmail.com>
 2021-09-12 18:24:15 +02:00
Adam Warner
4ac4bc5c18
Merge branch 'development' into fix/gravity_no_locallist 2021-07-04 00:48:09 +01:00
jpgpi250
a2d5b8050a Update gravity.sh 
Signed-off-by: jpgpi250 <jpgpi250@gmail.com>
 2021-06-14 20:27:10 +02:00
DL6ER
f90dffbf1a
Do not put hostname and pi.hole into local.list - this will be handled by FTL after FTL#1111 has been merged 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2021-04-16 11:41:26 +02:00
DL6ER
f536718aaa
Use possibly custom gravity.db location in the copying SQL script 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2021-03-18 16:46:59 +01:00
DL6ER
d25240fe9f
pihole -g should respose GRAVITYDB in pihole-FTL.conf 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2021-03-18 09:00:27 +01:00
DL6ER
126b9ae381
Add missing three new columns to pihole -g -r 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2021-03-02 20:08:09 +01:00
DL6ER
57fe3b6621
Merge pull request #3951 from pi-hole/new/gravity_adlist_infos 
Store gravity details in gravity.db adlist table
 2021-02-28 17:01:19 +01:00
Adam Warner
9ec012289f
Merge pull request #4017 from mwoolweaver/patch-2 
fix suspected typo in comment
 2021-01-26 08:48:05 +00:00
Michael Woolweaver
c42b97ddb2
fix typo in comment 
Signed-off-by: Michael Woolweaver <m.woolweaver@icloud.com>
 2021-01-26 01:04:37 -06:00
freddii
7c0c30fb0b fixed typos 2021-01-19 19:33:38 +01:00
DL6ER
1791fe22f6
Merge pull request #3912 from mdujava/basic_auth 
Fix validation of adlist url
 2020-12-31 12:19:32 +01:00
DL6ER
a2625df5e2
Remove "The list is new" comment because it is superfluous 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-12-30 11:27:34 +01:00
DL6ER
0944807491
Actually store correct number of domains for the individual lists (and not the sum of the so far collected number of domains) 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-12-29 20:36:49 +01:00
DL6ER
a216848c1d
Only update time of last list change when we see a list for the first time or when it really changed content 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-12-29 20:28:09 +01:00
DL6ER
8c56f54a1e
Compare checksum of downloaded list against older checksums to see if the list content changed since the last download 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-12-29 09:54:25 +01:00
DL6ER
6b33b8b4c0
Store status of downloaded list (downloaded, using cache, some error, ...) and number of (in-)valid domains on this list in the gravity database. This updates the gravity databaes to version 14. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-12-28 08:08:49 +01:00
Matej Dujava
ef0bdf6470 Fix validation of adlist url 
Already existing regex validation will be used on url after removing @ (in case
its in separating userinfo and host).

Signed-off-by: Matej Dujava <mdujava@kocurkovo.cz>
Fixes: https://github.com/pi-hole/pi-hole/issues/3911
Fixes: 7d19ee1b: validate blocklist URL before adding to the database (#3237)
 2020-12-21 12:50:02 +01:00
user
922f795fe6 Adding comma to the adlist table format 
To account for the Date Updated column in the adlist table adding a comma to the adlist to be imported into the database. This was causing intermittent failures on updates and therefore no domains would be blocked.

Signed-off-by: user <user@fedbrew.home>

 Date:      Sun Dec 6 06:03:30 2020 -0500
 Committer: user <user@fedbrew.home>

 On branch fix/gravity_date_updated
 Your branch and 'origin/fix/gravity_date_updated' have diverged,
 and have 1 and 1 different commits each, respectively.
   (use "git pull" to merge the remote branch into yours)

 Changes to be committed:
	modified:   gravity.sh

Signed-off-by: user <user@fedbrew.home>
 2020-12-06 06:08:30 -05:00
Adam Warner
948337dd8e
Merge pull request #3740 from pi-hole/new/adlist_date_updated 
Add date_updated field to adlist table
 2020-10-09 14:04:13 +01:00
DL6ER
4fd412d7c3
Add date_updated field in adlist table set when a list changes. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-09-07 20:59:23 +02:00
DL6ER
aa7c3b6852
Remove gravity optimization. Further investigations have shown that it is useless because the standard gravity run already produces an optimal database. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-09-07 20:50:11 +02:00
Adam Warner
bb1a6243d6
Merge pull request #3669 from pi-hole/release/v5.1.2 
Release/v5.1.2 to development
 2020-08-14 16:28:53 +01:00
Christian König
27399a762a Check for compression onyl once and print result 
Signed-off-by: Christian König <ckoenig@posteo.de>
 2020-08-03 22:46:14 +02:00
DL6ER
87da9084e6
Use compression (if available) when downloading the ad lists. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-08-02 16:21:58 +01:00
DL6ER
527fc3c5cf
Also pidof -> pgrep in gravity.sh 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-07-21 22:03:54 +02:00
Christian König
73963fecda Use gravity's adlist_id in filename when saving downloaded adlist locally 
Signed-off-by: Christian König <ckoenig@posteo.de>
 2020-07-05 14:32:33 +02:00
Peter Lowe
dd205c55c1 Updated gravity.sh to allow semicolons 
Signed-off-by: Peter Lowe <pgl@yoyo.org>
 2020-05-26 15:53:10 +02:00
Adam Warner
9e96ff37de
Fix merge conflicts 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2020-05-12 22:16:43 +01:00
DL6ER
288d487fc0
Allow ( and ) in adlist URLs. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-04-26 09:33:09 +02:00
DL6ER
fa57c457f3
Boldify number of unique domains as this is the actually interesting number 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-04-21 09:10:21 +02:00
Dan Schaper
0f20470a38
Merge pull request #3269 from pi-hole/tweak/hosts-comments 
Add support for comments in HOSTS-like files
 2020-04-15 11:11:10 -07:00
DL6ER
982c1b0059
Merge branch 'release/v5.0' into development 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-04-08 12:02:53 +02:00
Adam Warner
7d19ee1b25
validate blocklist URL before adding to the database (#3237) 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>

Co-authored-by: DL6ER <dl6er@dl6er.de>
 2020-03-31 21:48:10 +01:00
DL6ER
7b15a88dc4
Strip comments from downloaded lists instead of discarding lines with comments altogether 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-03-31 18:36:40 +00:00
XhmikosR
7b8611ced0
Assorted typo fixes. (#3126) 
* Assorted typo fixes.

Signed-off-by: XhmikosR <xhmikosr@gmail.com>

Co-authored-by: Dan Schaper <dan.schaper@pi-hole.net>
 2020-03-08 16:53:14 -07:00
DL6ER
3dd05606ca
Call it the received number of domains instead of the imported number as importing does only happen a bit later. Only show the number of invalid domains if there are invalid domains. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-24 07:06:15 +01:00
DL6ER
1e8bfd33f5
Improve output 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-23 22:50:06 +01:00
DL6ER
8131b5961c
Add comments to the code describing the changes. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-22 15:22:29 +01:00
DL6ER
81d4531e10
Implement performant list checking routine. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-22 13:01:55 +01:00
DL6ER
050e2963c7
Remove redundant code. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-21 22:28:53 +01:00
DL6ER
3c09cd4a3a
Experimental output of matching line from shown warnings. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-21 21:47:56 +01:00
DL6ER
839fe32042
Fix issue with missing newline at the end of adlists (#3144) 
* Also display non-fatal warnings during the database importing. Previously, we have only show warnings when there were also errors (errors are always fatal).

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Ensure there is always a newline on the last line.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Stickler linting

Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>

* Move sed command into subroutine to avoid code duplication.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Also unify comments.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Also unify comments.

Signed-off-by: DL6ER <dl6er@dl6er.de>

Co-authored-by: Dan Schaper <dan@glacialmagma.com>
 2020-02-21 18:56:48 +00:00
DL6ER
d14ee26d6a
Merge pull request #3139 from pi-hole/fix/count_before_calling_FTL 
Fix wrong number of blocking domains shown on the dashboard
 2020-02-17 21:32:45 +01:00
DL6ER
52398052e9
Compute number of domains (and store it in the database) BEFORE calling FTL to re-read said value. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-17 21:07:48 +01:00
DL6ER
601f9048cd
Merge pull request #3130 from pi-hole/fix/gravity_updated_timestamp 
Store gravity update timestamp only after database swapping
 2020-02-17 06:07:05 +01:00
DL6ER
dc2fce8e1d
Store gravity update timestamp only after database swapping. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-12 19:26:25 +01:00
DL6ER
2a5cf221fa
Store number of distinct gravity domains in database after counting. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-02 23:46:33 +01:00
DL6ER
10c2dad48a Improve gravity performance (#3100) 
* Gravity performance improvements.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Do not move downloaded lists into migration_backup directory.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Do not (strictly) sort domains. Random-leaf access is faster than always-last-leaf access (on average).

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Append instead of overwrite gravity_new collection list.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Rename table gravity_new to gravity_temp to clarify that this is only an intermediate table.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Add timers for each of the calls to compute intense parts. They are to be removed before this finally hits the release/v5.0 branch.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Fix legacy list files import. It currently doesn't work when the gravity database has already been updated to using the single domainlist table.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Simplify database_table_from_file(), remove all to this function for gravity lost downloads.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Update gravity.db.sql to version 10 to have newle created databases already reflect the most recent state.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Create second gravity database and swap them on success. This has a number of advantages such as instantaneous gravity updates (as seen from FTL) and always available gravity blocking. Furthermore, this saves disk space as the old database is removed on completion.

* Add timing output for the database swapping SQLite3 call.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Explicitly generate index as a separate process.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Remove time measurements.

Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-01-24 09:39:13 -08:00
DL6ER
8f22203d24
Wait 30 seconds for obtaining a database lock instead of immediately failing if the database is busy. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-01-14 20:02:00 +01:00
DL6ER
948f4a8827
Ensure permissions and ownership of gravity.db are correctly set on each run of pihole -g. This would have prevented https://github.com/pi-hole/AdminLTE/issues/1077 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-12-16 09:55:46 +00:00
DL6ER
2cec9eaf65
Merge pull request #3033 from pi-hole/fix/duplicates_in_adlists 
Remove duplicates from adlists before importing
 2019-12-12 21:37:26 +01:00
DL6ER
779fe670f7
Show full URL during gravity download instead of only domain and file 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-12-12 10:29:44 +00:00
DL6ER
570a7a5c11
Use sort -u instead of uniq as it is guaranteed to be safe when doing inline file operations. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-12-12 10:17:59 +00:00
DL6ER
bd1b004d94
Remove possible duplicates found in lower-quality adlists 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-12-12 10:13:51 +00:00
Adam Warner
d29947ba32
optimise gravity list inserts 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2019-12-09 22:30:41 +00:00
Adam Warner
1f03faddef
shell check recomends 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2019-12-09 21:35:54 +00:00
DL6ER
3231e5c3ba
Address stickler requests. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-12-09 16:52:03 +00:00