Commit Graph

478 Commits (9939cf1d774ced34b528c37ea6a90137419315b5)

Author SHA1 Message Date
DL6ER 1791fe22f6
Merge pull request #3912 from mdujava/basic_auth
3 years ago
DL6ER a2625df5e2
Remove "The list is new" comment because it is superfluous
3 years ago
DL6ER 0944807491
Actually store correct number of domains for the individual lists (and not the sum of the so far collected number of domains)
3 years ago
DL6ER a216848c1d
Only update time of last list change when we see a list for the first time or when it really changed content
3 years ago
DL6ER 8c56f54a1e
Compare checksum of downloaded list against older checksums to see if the list content changed since the last download
3 years ago
DL6ER 6b33b8b4c0
Store status of downloaded list (downloaded, using cache, some error, ...) and number of (in-)valid domains on this list in the gravity database. This updates the gravity databaes to version 14.
3 years ago
Matej Dujava ef0bdf6470 Fix validation of adlist url
3 years ago
user 922f795fe6 Adding comma to the adlist table format
3 years ago
Adam Warner 948337dd8e
Merge pull request #3740 from pi-hole/new/adlist_date_updated
4 years ago
DL6ER 4fd412d7c3
Add date_updated field in adlist table set when a list changes.
4 years ago
DL6ER aa7c3b6852
Remove gravity optimization. Further investigations have shown that it is useless because the standard gravity run already produces an optimal database.
4 years ago
Adam Warner bb1a6243d6
Merge pull request #3669 from pi-hole/release/v5.1.2
4 years ago
Christian König 27399a762a Check for compression onyl once and print result
4 years ago
DL6ER 87da9084e6
Use compression (if available) when downloading the ad lists.
4 years ago
DL6ER 527fc3c5cf
Also pidof -> pgrep in gravity.sh
4 years ago
Christian König 73963fecda Use gravity's adlist_id in filename when saving downloaded adlist locally
4 years ago
Peter Lowe dd205c55c1 Updated gravity.sh to allow semicolons
4 years ago
Adam Warner 9e96ff37de
Fix merge conflicts
4 years ago
DL6ER 288d487fc0
Allow ( and ) in adlist URLs.
4 years ago
DL6ER fa57c457f3
Boldify number of unique domains as this is the actually interesting number
4 years ago
Dan Schaper 0f20470a38
Merge pull request #3269 from pi-hole/tweak/hosts-comments
4 years ago
DL6ER 982c1b0059
Merge branch 'release/v5.0' into development
4 years ago
Adam Warner 7d19ee1b25
validate blocklist URL before adding to the database (#3237)
4 years ago
DL6ER 7b15a88dc4
Strip comments from downloaded lists instead of discarding lines with comments altogether
4 years ago
XhmikosR 7b8611ced0
Assorted typo fixes. (#3126)
4 years ago
DL6ER 3dd05606ca
Call it the received number of domains instead of the imported number as importing does only happen a bit later. Only show the number of invalid domains if there are invalid domains.
4 years ago
DL6ER 1e8bfd33f5
Improve output
4 years ago
DL6ER 8131b5961c
Add comments to the code describing the changes.
4 years ago
DL6ER 81d4531e10
Implement performant list checking routine.
4 years ago
DL6ER 050e2963c7
Remove redundant code.
4 years ago
DL6ER 3c09cd4a3a
Experimental output of matching line from shown warnings.
4 years ago
DL6ER 839fe32042
Fix issue with missing newline at the end of adlists (#3144)
4 years ago
DL6ER d14ee26d6a
Merge pull request #3139 from pi-hole/fix/count_before_calling_FTL
4 years ago
DL6ER 52398052e9
Compute number of domains (and store it in the database) BEFORE calling FTL to re-read said value.
4 years ago
DL6ER 601f9048cd
Merge pull request #3130 from pi-hole/fix/gravity_updated_timestamp
4 years ago
DL6ER dc2fce8e1d
Store gravity update timestamp only after database swapping.
4 years ago
DL6ER 2a5cf221fa
Store number of distinct gravity domains in database after counting.
4 years ago
DL6ER 10c2dad48a Improve gravity performance (#3100)
4 years ago
DL6ER 8f22203d24
Wait 30 seconds for obtaining a database lock instead of immediately failing if the database is busy.
4 years ago
DL6ER 948f4a8827
Ensure permissions and ownership of gravity.db are correctly set on each run of pihole -g. This would have prevented https://github.com/pi-hole/AdminLTE/issues/1077
4 years ago
DL6ER 2cec9eaf65
Merge pull request #3033 from pi-hole/fix/duplicates_in_adlists
4 years ago
DL6ER 779fe670f7
Show full URL during gravity download instead of only domain and file
4 years ago
DL6ER 570a7a5c11
Use sort -u instead of uniq as it is guaranteed to be safe when doing inline file operations.
4 years ago
DL6ER bd1b004d94
Remove possible duplicates found in lower-quality adlists
4 years ago
Adam Warner d29947ba32
optimise gravity list inserts
5 years ago
Adam Warner 1f03faddef
shell check recomends
5 years ago
DL6ER 3231e5c3ba
Address stickler requests.
5 years ago
DL6ER 0c5185f8ba
Also display how many unique domains we have caught in the event horizon.
5 years ago
DL6ER 037d52104a
New command "pihole -g -r" recreates gravity.db based on files backed up in /etc/pihole/migration_update. This is useful to restore a working version of the database when the user destroyed the original database. Also, update gravity.db to version 5 because of a fix we needed to implement.
5 years ago
DL6ER 756c99653e
Merge branch 'development' into new/internal-blocking
5 years ago
Adam Warner 61a40c1b43
merge devel into 4.3.2 And Resolve merge conflicts
5 years ago
DL6ER a8af2e1837
Store domains without sorting and unifying them first. This allows us to preserve the relationship of the individual domains to the lists they came from.
5 years ago
Adam Warner 8bef5dc805
remove n from -ne
5 years ago
Adam Warner ad41bcca5a
Remove support for adblock style lists to prevent false positives
5 years ago
DL6ER ca8982494b
Store timestamp when the gravity table was last updated successfully. This fixes https://github.com/pi-hole/AdminLTE/issues/989
5 years ago
DL6ER 1820c2c598
Merge branch 'development' into new/whitelist-regex-support
5 years ago
Andreas c53be459c6
quick fix for when dig also returns a CNAME
5 years ago
DL6ER 4371c9ba03
Ensure proper permissions are set for gravity.db after creation.
5 years ago
Andreas 63230cb72d
quick fix for when dig also returns a CNAME
5 years ago
DL6ER 5ff9052200
Review comments
5 years ago
DL6ER f5121c64be
We should still add the regex lines (initially) to the regex table as the renaming will happen only after the importing.
5 years ago
DL6ER 054c7a2c05
Create new table + view regex_whitelist + rename old regex table to regex_blacklist. This updates the gravity.db version to 3.
5 years ago
DL6ER be3e198f9a
Address linting errors.
5 years ago
DL6ER acc50b709e
Only migrate files once (domain and adlist lists druing initial creation of gravity.db auditlog.list on database upgrade from version 1 to 2.
5 years ago
DL6ER 0405aaa3da
Review comments and fixing stickler complaints.
5 years ago
DL6ER 2fb4256f84
Rename table to "auditlist"
5 years ago
DL6ER 4f4a12bb40
Upgrade database if necessary and store audit domains therein.
5 years ago
DL6ER 7a1b6fd205
Merge branch 'development' into new/gravity.db_domain_groups
5 years ago
DL6ER f91421418a
Rename adlists -> adlist table.
5 years ago
DL6ER 87c115fc86
Change implementation to use linking tables (domain_groups, adlists_groups). This allows adding domains to multiple lists conveniently.
5 years ago
DL6ER d5e8f1a781
Add 10 seconds timeout to the sqlite3 command that writes tables to the gravity database. This prevents gravity from failing when the database file is locked for a short time.
5 years ago
DL6ER cbc210b014
Add adlist_groups and mark group_id columns as NOT NULL.
5 years ago
Chris Crocker-White 69dba022c4 Handle an empty local.list
5 years ago
Dan Schaper 7de6bbca61
Merge branch 'development' into release/v4.3
5 years ago
Mcat12 8e1bf6e3d4
Merge branch 'development' into new/gravitydb
5 years ago
pvogt09 3f90261520 Moves chmod after every creation of ${destination} instead of end of function
5 years ago
pvogt09 225285cb97 Removes chmod for ${source} and superfluous chmod for ${destination}
5 years ago
pvogt09 a3d2a1062f Makes blocklists world readable
5 years ago
Mcat12 ec06fd0ad4
Fix incorrect adlist query when an adlist is blocked during gravity
5 years ago
DL6ER 0cfecd31d3
Invert logic to reduce nesting
5 years ago
DL6ER 008e88b84b
Only migrate legacy list files when we create gravity.db in the same gravity run
5 years ago
DL6ER 540c57f755
Add comment for automigrated list entries
5 years ago
DL6ER 983117d788
Add id (primary key, autoincrement) field to lists tables
5 years ago
DL6ER b372f808dd
Fix lint errors
5 years ago
pvogt09 8a92fb24c4 Incorporates changes requested by @Mcat12
5 years ago
pvogt09 5c575e73c7 Adds world readable attribute to files created by Pi-hole to circumvent #2724
5 years ago
DL6ER a932209143
Set default for date_modified to be the current time
5 years ago
DL6ER 4946b5907b
Move migrated files to backup directory instead of deleting them. This has the benefit that gravity.db can be recreated at any time by: 1. deleting gravity.db, 2. copying all files from the migration_backup directory back to /etc/pihole, 3. running pihole -g again.
5 years ago
Mark Drobnak 4596d9a577
Merge branch 'development' into new/gravitydb
5 years ago
DL6ER 8524aecfed
Don't set gravity database permissions
5 years ago
DL6ER 3dcb9722d3
Remove global haveSourceUrls variable and just skip downloading and processing adlists if there aren't any to be downloaded
5 years ago
DL6ER bd0215524b
Add vw_adlists view which returns only enabled adlists
5 years ago
DL6ER 53e3ff2b24
Only add non-empty lines when moving file content into database tables
5 years ago
DL6ER 0efc46260f
Use both old and new strings for detecting whether this is a Pi-hole dnsmasq config file
5 years ago
DL6ER d94bbfca7f
Finish migrating gravity.db to using our new gravity database. This removes the --skip-download, --blacklist-only, --whitelist-only, and --wildcard-only options as it is not necessary to run gravity when modifying them from now on (gravity really only builds the gravity database and the local.list file).
5 years ago
DL6ER 465a39427b
Source adlists from gravity database
5 years ago
DL6ER 2664ac0efc
Improve error reporting
5 years ago
DL6ER c1277705d8
Copy existing whitelist.txt, blacklist.txt, regex.list, and adlists.list to the database. We remove the files afterwards as the content lives in the database now
5 years ago
DL6ER e57d0fb93c
Remove the empty templates gravity.list, white.list, black.list. The old source files whitelist.txt, blacklist.txt, and regex.list still exist and need to be removed in a follow-up commit
5 years ago
DL6ER ee20164207
Remove detect_pihole_blocking_status() as we can modify the database at any time
5 years ago