README Add souin description

.gitignore

@@ -22,8 +22,5 @@ node_modules/
 .env
 .env.*
 
-# Ignore generated secrets
-dynamic.d/middlewares/crowdsec.yml
-
 # Ignore backup files
-*.~*    
+*.~*    

README.md

@@ -1,75 +1,93 @@
-# Traefik Edge Stack
+# GB Traefik Setup
 
-Reverse proxy for everything on this host. The goal: keep Cloudflare in front, expose a private LAN entrypoint, and let Docker stacks self-register through labels without leaking secrets.
+This repository contains the configuration files and setup instructions for deploying [Traefik](https://traefik.io/), a modern reverse proxy and load balancer.
 
-## Architecture Snapshot
+Configuration files is customized for Gbanyan personal usage. 
 
-- **Static config (`traefik.yml`)**  
-  - EntryPoints: `web/websecure` on `10.0.0.225`, `internal_web/internal_websecure` on `192.168.50.4`.  
-  - Trusted IP lists are managed by `scripts/update_cloudflare_ips.py`.  
-  - Docker provider is discovery-only; every container opts in with labels.  
-  - File provider loads everything in `dynamic.d/`.
+## Prerequisites
 
-- **Dynamic config (`dynamic.d/`)**  
-  - `middlewares/` – retry, compression, CrowdSec (rendered from template).  
-  - `transports/fast-upstreams.yml` – shared connection pool tuning.  
-  - `routers/` – internal-only routers (public ones stay in labels).
+- Docker installed on your system
+- Docker Compose (if using `docker-compose.yml`)
 
-- **Host networking**  
-  Traefik runs with `network_mode: host` so it can bind to both IPs simultaneously. Switching to bridge mode would require duplicating Traefik or adding another L4 hop, so host mode stays.
+## Getting Started
 
-## Secrets Workflow
+1. Clone this repository:
+    ```bash
+    git clone https://gitea.gbanyan.net/gbanyan/GB-Traefik.git
+    cd GB-Traefik
+    ```
 
-1. Copy `.env.example` → `.env` and fill:
-   - `CLOUDFLARE_EMAIL`, `CLOUDFLARE_DNS_API_TOKEN`
-   - `CROWDSEC_LAPI_KEY`
-2. Render secret-aware dynamic files:
-   ```bash
-   ./scripts/render_dynamic.sh
-   ```
-   This uses `templates/crowdsec.yml.tmpl` and writes `dynamic.d/middlewares/crowdsec.yml` (ignored by git).
+2. Update the `traefik.yml` and `docker-compose.yml` files as needed for your environment.
 
-## Runbook
+3. Start Traefik:
+    ```bash
+    docker compose up -d
+    ```
 
-```bash
-# start / update Traefik
-docker compose up -d traefik
+4. Access the Traefik dashboard (if enabled) at `http://<your-domain-or-ip>:8080`.
 
-# refresh Cloudflare IPs and restart safely
-python scripts/update_cloudflare_ips.py
+## Configuration
 
-# tail logs
-tail -f traefik.log
-tail -f access.log
+- **.env**: Cloudflare E-mail and API Token for SSL DNS Challenge
+- **Traefik Configuration**: Modify `traefik.yml`, `dynamic.yml` to customize Traefik's behavior.
+- **Docker Compose**: Use `docker-compose.yml` to define services and networks.
+
+
+## Detail: 
+
+My traefik is split into internal and external entrypoint. 
+
+Internal entrypoint is for private and secure service without exposing. 
+
+Each entrypoint is binded to different ip address for isolation. 
+
+Then, other docker service is attached to different entrypoint guided by label in docker compose
+
+```yaml
+label: 
+    - "traefik.http.routers.service-name.entrypoints=websecure"
 ```
 
-Rotate CrowdSec keys? Edit `.env`, rerun `render_dynamic.sh`, then `docker compose up -d traefik`.
+Besides the entrypoint setup, I add cloudflare proxy (for exposing real ip to access.log for crowdsec to read), crowdsec-firewall-bouncer, compression with brotli middlrewares method in traefik.yml and dynamic.yml
 
-## Service Labels Cheat Sheet
+Adding middlewares is also guided by labels: 
+
+```yaml
+label: 
+    - "traefik.http.routers.service-name.middlewares=cloudflarewarp@file,crowdsec@file,compress-middleware@file"
+```
+
+The order of middlewares is meaningful. 
+
+Traefik has ability to apply SSL certs automatically. 
+Just offer the required DNS API authentication (Like cloudflare). 
+
+Please refer the traefik documentation. 
+
+The following is an example of a docker service I hosted in its docker-compose.yaml: 
 
 ```yaml
 labels:
-  - traefik.enable=true
-  - traefik.http.routers.myapp.rule=Host(`app.example.com`)
-  - traefik.http.routers.myapp.entrypoints=websecure
-  - traefik.http.routers.myapp.tls.certresolver=letsencrypt
-  - traefik.http.routers.myapp.middlewares=crowdsec@file,retry-fast@file,compress-middleware@file
-  - traefik.http.services.myapp.loadbalancer.serversTransport=fast-upstreams@file
-  - traefik.http.services.myapp.loadbalancer.server.port=3000
-  - traefik.docker.network=traefik_default
+      - "traefik.enable=true"
+      - "traefik.http.routers.ghost.entrypoints=websecure"
+      - "traefik.http.routers.ghost.rule=Host(`blog.gbanyan.net`)"
+      - "traefik.http.services.ghost.loadbalancer.server.port=2368"
+      - "traefik.http.routers.ghost.tls.certresolver=letsencrypt"
+      - "traefik.http.routers.ghost.middlewares=cloudflarewarp@file,crowdsec@file,compress-middleware@file"
+      - "com.centurylinklabs.watchtower.enable=true"
+      - "traefik.docker.network=traefik_default"
 ```
 
-Most stacks use the same middleware chain: CrowdSec bouncer (plugin), retry, and compression. Internal-only services skip CrowdSec by pointing at the `internal_*` entrypoints.
+I mount the access.log for crowdsec firewall to read. 
 
-## Performance Notes
+PS: Because I access my traefik dashboard through my local network. I commented out the authetication method for dashboard. 
 
-- Access logs are buffered with headers trimmed to keep syscalls down.  
-- Compression enforces a 1 KB minimum and respects the client’s preferred encoding.  
-- Shared transport keeps 64 idle connections per backend with aggressive idle/response timeouts.  
-- `retry-fast` retries once after 50 ms, smoothing transient Puma/Node hiccups without hammering backends.
+## Discussion and Changelog
 
-## Things to Remember
+1. Traefik vs Nginx
+- Performance: Nginx is still better at high traffic. After all it is written in C. Traefik 3 though claims it has higher 20% performance than before. The latency still showed a little higher than nginx. 
+- Docker Deployment Ease: Traefik is easier for docker service deployment. In my environment, I can assign each docker stack with labels and then guides the traefik to add Let's encrypt SSL. 
 
-- Watchtower is still enabled for Traefik; pin the image tag when you need deterministic upgrades.  
-- `scripts/update_cloudflare_ips.py` rewrites the static trusted IP block and restarts Traefik—run it via cron.  
-- Dashboard auth is intentionally disabled because access only happens from the LAN entrypoint. If that changes, re-enable `basicauth`.
+2. ChangeLog: 
+
+- 2025.4.18 Add Souin HTTP Cache Middleware. 

docker-compose.yaml

@@ -6,7 +6,7 @@ services:
 #    ports:
 #       - 10.0.0.225:80:80
 #       - 10.0.0.225:443:443
-#       - 192.168.50.4:8080:8080
+#       - 192.168.50.4:9090:9090
 #       - 192.168.50.4:80:80
 #       - 192.168.50.4:443:443 # Added port mapping for the dashboard
     restart: unless-stopped
@@ -17,7 +17,7 @@ services:
       - /var/run/docker.sock:/var/run/docker.sock:ro
       - ./certs:/letsencrypt
       #- ./dashboard_authfile:/dashboard_authfile:ro
-      - ./dynamic.d:/dynamic.d
+      - ./dynamic.yml:/dynamic.yml
       - ./traefik.yml:/traefik.yml
       - ./traefik.log:/var/log/traefik/traefik.log
       - ./access.log:/var/log/traefik/access.log
@@ -33,17 +33,15 @@ services:
       - "com.centurylinklabs.watchtower.enable=true" # Added label for Watchtower
       # "traefik.http.middlewares.auth.basicauth.usersfile=/dashboard_authfile"
       - "traefik.http.services.traefik.loadbalancer.server.port=9090"
-
-#networks:
+  redis:
+    image: valkey/valkey:latest
+    container_name: traefik-redis
+    restart: unless-stopped
+    networks:
+      internal_traefik_default:
+        ipv4_address: 172.20.0.100
+networks:
 #   traefik_default:
 #     external: true
-#   internal_traefik_default:
-#     external: true
-networks:
-  default:
-    name: traefik_default
-    driver: bridge
-    ipam:
-      config:
-        - subnet: 172.19.0.0/16
-          gateway: 172.19.0.1
+  internal_traefik_default:
+    external: true

dynamic.d/middlewares/compress.yml

@@ -1,15 +0,0 @@
-http:
-  middlewares:
-    compress-middleware:
-      compress:
-        encodings:
-          - zstd
-          - br
-          - gzip
-        minResponseBodyBytes: 1024
-        includedContentTypes:
-          - text/html
-          - text/css
-          - application/javascript
-          - application/json
-          - text/plain

dynamic.d/middlewares/retry.yml

@@ -1,6 +0,0 @@
-http:
-  middlewares:
-    retry-fast:
-      retry:
-        attempts: 2
-        initialInterval: 50ms

dynamic.d/routers/internal.yml

@@ -1,14 +0,0 @@
-http:
-  routers:
-    qbit:
-      rule: Host(`qbit.gbanyan.net`)
-      service: qbit
-      entryPoints:
-        - internal_websecure
-      tls:
-        certResolver: letsencrypt
-  services:
-    qbit:
-      loadBalancer:
-        servers:
-          - url: "http://192.168.50.4:8083"

dynamic.d/transports/fast-upstreams.yml

@@ -1,7 +0,0 @@
-http:
-  serversTransports:
-    fast-upstreams:
-      maxIdleConnsPerHost: 64
-      forwardingTimeouts:
-        idleConnTimeout: 30s
-        responseHeaderTimeout: 15s

dynamic.yml

@@ -0,0 +1,70 @@
+http:
+  middlewares:
+    block-ip-access:
+      headers:
+        customRequestHeaders:
+          Host: ""  # This will catch requests with no Host header or invalid ones
+    cloudflarewarp:
+      plugin:
+        cloudflare:
+          trustedCIDRs: []
+          overwriteRequestHeader: true
+          debug: true
+    crowdsec:
+      plugin:
+        bouncer:
+          enabled: true
+          crowdsecMode: stream
+          crowdsecLapiHost: "localhost:8080"
+          crowdsecLapiKey: gFJjSzdbB0GCe/1Y9HcxMPP1vQmoa4psZOFyleJZJVQ
+    compress-middleware:
+      compress:
+        encodings:
+          - zstd
+          - br
+          - gzip
+        defaultEncoding: zstd
+        includedContentTypes:
+          - text/html
+          - text/css
+          - application/javascript
+          - application/json
+          - text/plain
+    http-cache:
+      plugin:
+        souin:
+          default_cache:
+            ttl: 10s
+            default_cache_control: public, max-age=600
+            redis:
+              url: 172.20.0.100://redis:6379
+            allowed_http_verbs:
+              - GET
+              - HEAD
+              - POST
+          log_level: debug
+          api:
+            souin: {}
+            prometheus: {}
+  routers:
+    block-direct-access:
+      rule: "HostRegexp(`{host:.+}`)"  # Matches any host
+      service: noop@internal
+      priority: 1  # Low priority to catch unmatched requests
+      entryPoints:
+        - web
+        - websecure
+      middlewares:
+        - block-ip-access
+    netdata:
+      rule: Host(`netdata.gbanyan.net`)
+      service: netdata
+      entryPoints: ["internal_websecure"]
+      tls:
+        certResolver: letsencrypt
+
+  services:
+    netdata:
+      loadBalancer:
+        servers:
+          - url: "http://127.0.0.1:19999"

scripts/render_dynamic.sh

@@ -1,19 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
-cd "$ROOT_DIR"
-if [[ ! -f .env ]]; then
-  echo "Missing .env file. Copy .env.example and fill in secrets." >&2
-  exit 1
-fi
-set -a
-# shellcheck disable=SC1091
-source .env
-set +a
-: "${CROWDSEC_LAPI_KEY:?CROWDSEC_LAPI_KEY must be set in .env}"
-if ! command -v envsubst >/dev/null 2>&1; then
-  echo "envsubst is required to render templates." >&2
-  exit 1
-fi
-envsubst < templates/crowdsec.yml.tmpl > dynamic.d/middlewares/crowdsec.yml
-echo "Rendered dynamic.d/middlewares/crowdsec.yml"

scripts/update_cloudflare_ips.py

@@ -1,107 +0,0 @@
-#!/usr/bin/env python3
-"""Refresh Cloudflare IP ranges in Traefik config and restart Traefik if needed."""
-from __future__ import annotations
-
-import hashlib
-import subprocess
-import sys
-from pathlib import Path
-from typing import Iterable, List
-from urllib.request import Request, urlopen
-
-CLOUDFLARE_IPV4_URL = "https://www.cloudflare.com/ips-v4"
-CLOUDFLARE_IPV6_URL = "https://www.cloudflare.com/ips-v6"
-REPO_ROOT = Path(__file__).resolve().parents[1]
-TRAEFIK_CONFIG_PATH = REPO_ROOT / "traefik.yml"
-ENTRYPOINTS = ["  web:", "  websecure:"]
-USER_AGENT = "TraefikCloudflareUpdater/1.0 (+https://gbanyan.net)"
-
-
-def fetch_ip_ranges() -> List[str]:
-    ips: List[str] = []
-    for url in (CLOUDFLARE_IPV4_URL, CLOUDFLARE_IPV6_URL):
-        req = Request(url, headers={"User-Agent": USER_AGENT})
-        with urlopen(req, timeout=10) as resp:  # noqa: S310 (trusted upstream)
-            for raw_line in resp.read().decode().splitlines():
-                line = raw_line.strip()
-                if line and not line.startswith("#"):
-                    ips.append(line)
-    # Remove duplicates while preserving order
-    seen = set()
-    deduped: List[str] = []
-    for ip in ips:
-        if ip not in seen:
-            deduped.append(ip)
-            seen.add(ip)
-    return deduped
-
-
-def update_trusted_ips(original: str, ips: Iterable[str], entrypoint_label: str) -> str:
-    try:
-        entry_idx = original.index(entrypoint_label)
-    except ValueError:
-        raise RuntimeError(f"EntryPoint {entrypoint_label.strip()} not found in traefik.yml") from None
-
-    forwarded_token = "forwardedHeaders:\n"
-    fwd_idx = original.index(forwarded_token, entry_idx)
-    line_start = original.rfind("\n", 0, fwd_idx) + 1
-    indent = original[line_start:fwd_idx]
-    inner_indent = indent + "  "
-    list_indent = inner_indent + "  "
-
-    block_start = fwd_idx + len(forwarded_token)
-    cursor = block_start
-    while cursor < len(original):
-        next_newline = original.find("\n", cursor)
-        if next_newline == -1:
-            next_newline = len(original)
-        line = original[cursor:next_newline]
-        stripped = line.lstrip()
-        current_indent = len(line) - len(stripped)
-        if not stripped.startswith("-") and current_indent <= len(indent):
-            break
-        cursor = next_newline + 1
-    block_end = cursor
-
-    new_block = forwarded_token
-    new_block += f"{inner_indent}trustedIPs:\n"
-    for ip in ips:
-        new_block += f"{list_indent}- \"{ip}\"\n"
-
-    return original[:fwd_idx] + new_block + original[block_end:]
-
-
-def main() -> None:
-    ips = fetch_ip_ranges()
-    config_text = TRAEFIK_CONFIG_PATH.read_text()
-    before_hash = hashlib.sha256(config_text.encode()).hexdigest()
-
-    for entry_label in ENTRYPOINTS:
-        config_text = update_trusted_ips(config_text, ips, entry_label)
-
-    after_hash = hashlib.sha256(config_text.encode()).hexdigest()
-    if after_hash == before_hash:
-        print("Traefik trusted IP ranges already up-to-date.")
-        return
-
-    TRAEFIK_CONFIG_PATH.write_text(config_text)
-    print(f"Updated {TRAEFIK_CONFIG_PATH} with {len(ips)} Cloudflare IP ranges.")
-
-    try:
-        subprocess.run(
-            ["docker", "compose", "up", "-d", "traefik"],
-            check=True,
-            cwd=REPO_ROOT,
-        )
-        print("Traefik container restarted via docker compose.")
-    except subprocess.CalledProcessError as exc:  # pragma: no cover
-        print("Failed to restart Traefik via docker compose:", exc, file=sys.stderr)
-        sys.exit(exc.returncode)
-
-
-if __name__ == "__main__":
-    try:
-        main()
-    except Exception as exc:  # noqa: BLE001
-        print(f"Error: {exc}", file=sys.stderr)
-        sys.exit(1)

templates/crowdsec.yml.tmpl

@@ -1,9 +0,0 @@
-http:
-  middlewares:
-    crowdsec:
-      plugin:
-        bouncer:
-          enabled: true
-          crowdsecMode: stream
-          crowdsecLapiHost: "localhost:8080"
-          crowdsecLapiKey: "${CROWDSEC_LAPI_KEY}"

traefik.yml

@@ -1,27 +1,14 @@
 ## STATIC CONFIGURATION
 
 log:
-  level: "INFO"
+  level: "DEBUG"
   filePath: "/var/log/traefik/traefik.log"
 accessLog:
   filePath: "/var/log/traefik/access.log"
-  bufferingSize: 256
   filters:
     statusCodes:
-      - "400-599" # focus on failed http requests
-  fields:
-    defaultMode: drop
-    names:
-      ClientHost: keep
-      ClientPort: keep
-      Duration: keep
-      RequestMethod: keep
-      RequestPath: keep
-      RequestProtocol: keep
-      RouterName: keep
-      ServiceName: keep
-      ServiceURL: keep
-      Status: keep
+      - "200-299" # log successful http requests
+      - "400-599" # log failed http requests
 
 api:
   insecure: false
@@ -29,32 +16,34 @@ api:
 
 entryPoints:
   web:
-    address: "10.0.0.225:80"
-    forwardedHeaders:
-      trustedIPs:
-        - "173.245.48.0/20"
-        - "103.21.244.0/22"
-        - "103.22.200.0/22"
-        - "103.31.4.0/22"
-        - "141.101.64.0/18"
-        - "108.162.192.0/18"
-        - "190.93.240.0/20"
-        - "188.114.96.0/20"
-        - "197.234.240.0/22"
-        - "198.41.128.0/17"
-        - "162.158.0.0/15"
-        - "104.16.0.0/13"
-        - "104.24.0.0/14"
-        - "172.64.0.0/13"
-        - "131.0.72.0/22"
-        - "2400:cb00::/32"
-        - "2606:4700::/32"
-        - "2803:f800::/32"
-        - "2405:b500::/32"
-        - "2405:8100::/32"
-        - "2a06:98c0::/29"
-        - "2c0f:f248::/32"
-    http:
+   address: "10.0.0.225:80"
+   forwardedHeaders:
+      trustedIPs: &trustedIps
+        # Start of Cloudlare's public IP list
+        - 103.21.244.0/22
+        - 103.22.200.0/22
+        - 103.31.4.0/22
+        - 104.16.0.0/13
+        - 104.24.0.0/14
+        - 108.162.192.0/18
+        - 131.0.72.0/22
+        - 141.101.64.0/18
+        - 162.158.0.0/15
+        - 172.64.0.0/13
+        - 173.245.48.0/20
+        - 188.114.96.0/20
+        - 190.93.240.0/20
+        - 197.234.240.0/22
+        - 198.41.128.0/17
+        - 2400:cb00::/32
+        - 2606:4700::/32
+        - 2803:f800::/32
+        - 2405:b500::/32
+        - 2405:8100::/32
+        - 2a06:98c0::/29
+        - 2c0f:f248::/32
+        # End of Cloudlare's public IP list
+   http:
       redirections:                           # HTTPS redirection (80 to 443)
         entryPoint:
           to: "websecure"                         # The target element
@@ -62,29 +51,9 @@ entryPoints:
   websecure:
     address: "10.0.0.225:443"
     forwardedHeaders:
-      trustedIPs:
-        - "173.245.48.0/20"
-        - "103.21.244.0/22"
-        - "103.22.200.0/22"
-        - "103.31.4.0/22"
-        - "141.101.64.0/18"
-        - "108.162.192.0/18"
-        - "190.93.240.0/20"
-        - "188.114.96.0/20"
-        - "197.234.240.0/22"
-        - "198.41.128.0/17"
-        - "162.158.0.0/15"
-        - "104.16.0.0/13"
-        - "104.24.0.0/14"
-        - "172.64.0.0/13"
-        - "131.0.72.0/22"
-        - "2400:cb00::/32"
-        - "2606:4700::/32"
-        - "2803:f800::/32"
-        - "2405:b500::/32"
-        - "2405:8100::/32"
-        - "2a06:98c0::/29"
-        - "2c0f:f248::/32"
+      # Reuse the list of Cloudflare's public IPs from above
+      trustedIPs: *trustedIps
+    http3: {}
   internal_web:
     address: "192.168.50.4:80"
     http:
@@ -94,8 +63,9 @@ entryPoints:
             scheme: "https"
   internal_websecure:
     address: "192.168.50.4:443"
+    http3: {}
   metrics:
-    address: ":8082"
+    address: "127.0.0.1:8082"
   dashboard:
     address: "127.0.0.1:9090"
 
@@ -106,9 +76,9 @@ global:
 providers:
   docker:
     exposedByDefault: false
-    # defaultRule: "Host(`{{ .ContainerName }}.gbanyan.net`)"
+#    network: traefik_default  # Ensure this matches the Docker network
   file:
-    directory: "/dynamic.d"
+    filename: "/dynamic.yml"  # Enable dynamic configuration file
 certificatesResolvers:
   letsencrypt:
     acme:
@@ -127,6 +97,12 @@ metrics:
 
 experimental:
   plugins:
+    cloudflare:
+      moduleName: github.com/agence-gaya/traefik-plugin-cloudflare
+      version: v1.2.0
     bouncer:
       moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
       version: v1.4.2
+    souin:
+      moduleName: github.com/darkweak/souin
+      version: v1.7.6