Commit Graph

98 Commits

Author SHA1 Message Date
Kwitsch debac9eaa8
Refactoring/linter (#1447)
* update golangci-lint to v1.57.2

* linter fixes
2024-04-12 16:44:50 -04:00
ThinkChaos bcd1381e18 feat: update list config and code to use "allow/deny" language 2024-04-11 18:33:09 -04:00
ThinkChaos d5b6ee93b5 fix: use proxy env vars via Go default HTTP Transport values
Don't build `http.Transport` instances from scratch, but start from
`http.DefaultTransport` and override what is needed.
2024-04-10 09:43:57 -04:00
Kwitsch 03131c443c
Refactoring/config struct names (#1315)
* vscode config changed in new version

* FilteringConfig -> Filtering

* CachingConfig -> Caching

* QueryLogConfig -> QueryLog

* MetricsConfig -> Metrics

* HostsFileConfig -> HostsFile

* PortsConfig -> Ports

* BootstrapDNSConfig -> BootstrapDNS

* BootstrappedUpstreamConfig -> BootstrappedUpstream

* bootstrapDNSConfig -> bootstrapDNS

* bootstrappedUpstreamConfig -> bootstrappedUpstream

* SourceLoadingConfig -> SourceLoading

* DownloaderConfig -> Downloader
2023-12-20 15:38:33 -05:00
ThinkChaos a6654dcd57 refactor(config): rename `StartStrategyType` to `InitStrategy` 2023-12-05 20:52:46 -05:00
ThinkChaos 8c7b89cbb3 refactor(tests): remove `TmpFile.Error` to centralize error checks 2023-12-01 15:53:11 -05:00
ThinkChaos 891d0fba74 refactor(tests): move `DeferCleanup` into helper functions 2023-12-01 15:53:11 -05:00
ThinkChaos f371857150 style(tests): use `ContainElements` where possible 2023-12-01 15:53:11 -05:00
ThinkChaos a330174949 style(tests): `ShouldNot(Succeed())` -> `Should(HaveOccurred())` 2023-12-01 15:53:11 -05:00
ThinkChaos 321680250a style(tests): `Should(Not(x))` -> `ShouldNot(x)` 2023-12-01 15:53:11 -05:00
Kwitsch 3378316982
Refactoring: FileDownloader (#1281)
* enabled containedctx & contextcheck

* enabled noctx

* less background context

* context metrics test

* use ginkgo context instead of background

* fix redis e2e tests

* made downloader context aware
2023-11-29 18:18:29 +01:00
ThinkChaos d52c598546
fix(tests): properly silence log output (#1259)
* fix(tests): properly silence log output

Using `init` allows it to also work for benchmarks.
And `log.Silence` was sometimes getting overridden by `log.init`.

* squash: fix(server): don't setup the logger again
2023-11-19 17:51:49 -05:00
ThinkChaos b498bc5094
feat(lists): add support for wildcard lists using a custom Trie (#1233) 2023-11-17 15:58:35 +01:00
Kwitsch 6c000090b1
Unit test refactoring & devcontainer enhancement(#1245)
* Whietlist dot imports for test packages

* improved generate-lcov script

* moved mock_call_sequence to helpertest

* removed default config for markdown

* generate seperate lcov.info for each directory

* path fix in lcov generation script

* ginkgo -> v2.13.1

* fixed redundant strings

* added exampleComKey

* added folder name to finish message

* hide lcov.info in devcontainer

* coverage plugin is bugged if lcov file is excluded

* moved all lcov files to coverage folder

* added requested toos #1251
2023-11-15 16:42:53 +01:00
Kwitsch 7615584944
Feature/ginkgolinter (#1235)
* add ginkgolinter to golangci config

* fixed ginkgolinter errors
2023-11-12 13:40:55 -05:00
Kwitsch 8ece708fe9
devcontainer & unit test refactoring (#1220)
* extension cleanup & added ginkgo watch

* added gcov2lcov

* added test explorer and reworked scripts

* go mod tidy

* use package cache volume

* script rework

* defined tasks

* defined launch

* don't try to convert if test was canceld

* generate lcov only in devcontainer

* disable coverage upload on forks

* wip: make lcov

* fixed unit tests for parallel

* parallel test for lists

* fix serve test for parallel

* parallel test fixes

* deleted accident commit

* wip: make lcov

* restructured settings location

* start script refactoring

* added GetProcessPort

* fixed parallel ports

* race fix

* changed port for github runner binding

* fixed local list var in test

* more local vars in tests fix

* less local vars

* run test & race parallel

* removed invalid error check

* fixed error  check

* less local variables

* fixed timing problem

* removed gcov2lcov

* added generate-lcov

* added GINKGO_PROCS to makefile

* fixed workflow

* run generate-lcov on save *.go

* added tooltitude
2023-11-11 11:58:31 -05:00
Dimitri Herzog 33ea933015
refactor: pass context for goroutine shutdown (#1187) 2023-10-07 16:21:40 -04:00
Dimitri Herzog 720e6550be chore(test): fix flaky test 2023-09-14 15:03:37 +02:00
Dimitri Herzog 72d747c16f
feat: API-first approach for REST interface (#1129)
* feat: embed OpenAPI definition file

* feat: use OpenAPI generated server and client

* feat: provide OpenAPI interface documentation

* chore(test): add additional tests
2023-09-09 19:30:55 +02:00
Dimitri Herzog 245bb613df
fix: don't convert regex from punycode to ASCII (#1126) (#1131)
* fix: don't convert regex from punycode to ASCII (#1126)

* chore(test): add additional tests
2023-09-08 18:11:26 +02:00
DerRockWolf c112e86740
feat: add upstream strategy `strict` (#1093) 2023-08-21 09:50:23 +02:00
ThinkChaos cfc3699ab5 feat: support multiple hosts files 2023-07-07 09:16:20 -04:00
ThinkChaos ea95d36f37 fix: make domain validation in list parser more lenient 2023-04-14 16:20:54 -04:00
ThinkChaos 8253075f28
test: remove focus in list_cache suite (#990)
Accidentally left `FIt` when committing the test.
2023-04-13 09:10:40 +02:00
mandrav 015b565137
Add maxErrorsPerFile blocking configuration (#986)
* Add maxErrorsPerFile blocking configuration

The default max errors per file of 5 is too small IMHO.
This commit makes this number user-configurable.

* squash: fix lint

* squash: docs

* squash: change type to int to allow -1

* squash: test that the `maxErrorsPerFile` is actually used

---------

Co-authored-by: ThinkChaos <ThinkChaos@users.noreply.github.com>
2023-04-12 20:43:49 +02:00
Kwitsch 68a8476e48
chore(build): Automation workflows (#972)
* Added fork sync workflow(syncs the main branch with upstream every 30 minutes)
* Added monthly workflow run deletion(all skipped or canceld and runs older than 30days will be deleted)
* Removed Docker image build for main branch on forks
* Added documentation how to enable sync and docker build workflows
2023-04-03 16:33:16 +02:00
ThinkChaos f887e82fdc fix: support IDNA in lists
Not sure this is actually supported by most adblocker, but we might as
well be permissive.
2023-03-29 10:05:50 -04:00
Dimitri Herzog 3b9fd7bafe
refactor: use groupedCache to optimize cache access (#944)
* refactor: use groupedCache to optimize cache access

* refactor: fix review findings
2023-03-27 13:23:01 +02:00
ThinkChaos 5088c75a78
refactor: configuration rework (usage and printing) (#920)
* refactor: make `config.Duration` a struct with `time.Duration` embed

Allows directly calling `time.Duration` methods.

* refactor(HostsFileResolver): don't copy individual config items

The idea is to make adding configuration options easier, and searching
for references straight forward.

* refactor: move config printing to struct and use a logger

Using a logger allows using multiple levels so the whole configuration
can be printed in trace/verbose mode, but only important parts are
shown by default.

* squash: rename `Cast` to `ToDuration`

* squash: revert `Duration` to a simple wrapper ("new type" pattern)

* squash: `Duration.IsZero` tests

* squash: refactor resolvers to rely on their config directly if possible

* squash: implement `IsEnabled` and `LogValues` for all resolvers

* refactor: use go-enum `--values` to simplify getting all log fields

* refactor: simplify `QType` unmarshaling

* squash: rename `ValueLogger` to `Configurable`

* squash: rename `UpstreamConfig` to `ParallelBestConfig`

* squash: rename `RewriteConfig` to `RewriterConfig`

* squash: config tests

* squash: resolver tests

* squash: add `ForEach` test and improve `Chain` ones

* squash: simplify implementing `config.Configurable`

* squash: minor changes for better coverage

* squash: more `UnmarshalYAML` -> `UnmarshalText`

* refactor: move `config.Upstream` into own file

* refactor: add `Resolver.Type` method

* squash: add `log` method to `typed` to use `Resolover.Type` as prefix

* squash: tweak startup config logging

* squash: add `LogResolverConfig` tests

* squash: make sure all options of type `Duration` use `%s`
2023-03-12 22:14:10 +01:00
ThinkChaos a2ab7c3ef1
feat: allow hosts file resolver to use a HTTP(S) link or inline block (#884)
Unify the hosts file parsing between the hosts resolver and lists so
the resolver supports more data sources than local files.

Lists' group cache is now re-used if refresh fails.

Also improve lookups in hosts:
Instead of iterating through all hosts+aliases for each A/AAAA query,
we can do a single lookup.
For PTR we search through only the hosts with an IP version that matches
the question. And compare IPs instead of building the reverse DNS name
for each IP in the hosts database.
2023-03-06 19:32:41 -05:00
Shizun Ge c199750b0d
chore(log): always report group name when cache populating failed. (#875)
Co-authored-by: ThinkChaos <ThinkChaos@users.noreply.github.com>
2023-02-17 08:57:27 +01:00
ThinkChaos 6548d15d8b fix: ignore end of line comments when parsing lists 2023-02-07 09:35:53 -05:00
Kwitsch 9abeaeddea
Added gofumpt (#805)
* fix linter errors

* added gofumpt

* Makefile cleanup

* gofumpt run

* Update Makefile

Co-authored-by: ThinkChaos <ThinkChaos@users.noreply.github.com>

* go mod tidy

* fmt + lint

* go mod tidy

Co-authored-by: ThinkChaos <ThinkChaos@users.noreply.github.com>
2022-12-26 22:11:45 +01:00
Dimitri Herzog 3c40ce5d3b
fix: no initial cache refresh with "fast" start strategy (#804) 2022-12-17 23:06:58 +01:00
ThinkChaos d501d867df style(resolver): standardize configuration disabled impls and tests 2022-12-04 20:59:12 -05:00
Dimitri Herzog f78a57a94d
chore(build): add additional linters (#774)
* chore(build): add nolintlint linter

* chore(build): add usestdlibvars linter
2022-11-29 21:58:26 +01:00
Dimitri Herzog 3e95b12eed feat: caching for empty DNS responses (#700) 2022-11-08 21:45:28 +01:00
Kwitsch fd93f67899
feat: new parameter "StartStrategy" (#566) (#636)
Blocky should start resolving DNS traffic as soon as possible
2022-09-03 22:12:07 +02:00
FileGo 377f4764fe
Enable start as long as at least one upstream resolver in group is reachable (#608)
* Enable start if one upstream resolver fails

* Will now check if upstream actually works

* Fixed default upstream in some tests

* Increase timeouts in some tests

* change default value of "StartVerifyUpstream" to false

Co-authored-by: Dimitri Herzog <dimitri.herzog@gmail.com>
2022-08-21 17:21:08 +02:00
Dimitri Herzog 421807fc22
chore: update golangci-lint (#631) 2022-08-19 22:04:35 +02:00
Dimitri Herzog 3c5b7a20dc
chore: Managing of external tools with go modules (#616) (#617) 2022-08-03 22:10:07 +02:00
Dimitri Herzog b1973702bd chore: update golangci-lint to v1.47.3 2022-08-01 23:19:35 +02:00
Kwitsch 51ef9ea77f
test: fix flaky test(537) - second hunt (#610)
* go test -> ginkgo

* removed flake-attempts as suggested

* added FlakeAttempts to flaky test definition

* as net.DNSError qualifies as net.Error check and correct the response

* fix in downloader.go resolves flaky tests

* unwrap maybe?

* excluede DownloadFile from funlen linter

* use eventually on io operations in file_writer_test

* file_writer_test flaky fixes

* fix linter errors

* Serve test reversed go routine

* matcher fix

* serve test rework 2

* DeferCleanup tmp files

* fixed some flakiness in resolver tests

* go mod tidy

* fixed linter errors

* JustAfterEach -> DeferCleanup

* changed close to defer

* moved unwrap from downloader to test

* fix linter error

* propper cleanup in "should return error"

Co-authored-by: Dimitri Herzog <dimitri.herzog@gmail.com>
2022-08-01 22:27:11 +02:00
Kwitsch 061419f68f
chore: use ginkgo for test executions (#537) (#562)
* go test -> ginkgo

* added FlakeAttempts to flaky test definition

Co-authored-by: Dimitri Herzog <dimitri.herzog@gmail.com>
2022-07-08 08:04:08 +02:00
Dimitri Herzog 3adeea6691 chore: added debug output 2022-06-07 15:26:52 +02:00
FileGo 2944debf0b
Update to Go 1.18 (#540)
* Update to Go 1.18

* remove "nolint:unparam", since unparam is currently disabled with go 1.18

* update golangci-lint to 1.46.2

Co-authored-by: Dimitri Herzog <dimitri.herzog@gmail.com>
2022-05-27 22:41:11 +02:00
FileGo 8bb4f73a38
Removed net.Error.Temporary (#536) 2022-05-27 22:22:00 +02:00
Dimitri Herzog 3392de0e16
Cache refresh memory consumption optimization (#470) (#515) 2022-05-16 21:32:16 +02:00
Dimitri Herzog a4b89537db
update golangci-lint (#510)
* update golangci-lint

* enable gomnd linter

* enable asciicheck linter

* enable bidichk linter

* enable durationcheck linter

* enable errchkjson linter

* enable errorlint linter

* enable exhaustive linter

* enable gomoddirectives linter

* enable gomodguard guard

* enable grouper linter

* enable grouper and ifshort linters

* enable importas linter

* enable makezero linter

* enable nestif linter

* enable nilerr linter

* enable nilnil linter

* enable nlreturn linter

* enable nolintlint linter

* enable predeclared linter

* enable sqlclosecheck linter

* enable tenv linter

* enable wastedassign linter
2022-05-10 09:09:50 +02:00
Dimitri Herzog 53814a2208
chore: refactoring list cache, extracting download functionality (#508) 2022-05-06 17:57:33 +02:00