Revert Ghost router split

.gitignore

@@ -22,5 +22,8 @@ node_modules/
 .env
 .env.*
 
+# Ignore generated secrets
+dynamic.d/middlewares/crowdsec.yml
+
 # Ignore backup files
 *.~*    

README.md

@@ -1,105 +1,75 @@
-# GB Traefik Setup
+# Traefik Edge Stack
 
-This repository contains the configuration files and setup instructions for deploying [Traefik](https://traefik.io/), a modern reverse proxy and load balancer.
+Reverse proxy for everything on this host. The goal: keep Cloudflare in front, expose a private LAN entrypoint, and let Docker stacks self-register through labels without leaking secrets.
 
-Configuration files is customized for Gbanyan personal usage. 
+## Architecture Snapshot
 
-## Prerequisites
+- **Static config (`traefik.yml`)**  
+  - EntryPoints: `web/websecure` on `10.0.0.225`, `internal_web/internal_websecure` on `192.168.50.4`.  
+  - Trusted IP lists are managed by `scripts/update_cloudflare_ips.py`.  
+  - Docker provider is discovery-only; every container opts in with labels.  
+  - File provider loads everything in `dynamic.d/`.
 
-- Docker installed on your system
-- Docker Compose (if using `docker-compose.yml`)
+- **Dynamic config (`dynamic.d/`)**  
+  - `middlewares/` – retry, compression, CrowdSec (rendered from template).  
+  - `transports/fast-upstreams.yml` – shared connection pool tuning.  
+  - `routers/` – internal-only routers (public ones stay in labels).
 
-## Getting Started
+- **Host networking**  
+  Traefik runs with `network_mode: host` so it can bind to both IPs simultaneously. Switching to bridge mode would require duplicating Traefik or adding another L4 hop, so host mode stays.
 
-1. Clone this repository:
+## Secrets Workflow
+
+1. Copy `.env.example` → `.env` and fill:
+   - `CLOUDFLARE_EMAIL`, `CLOUDFLARE_DNS_API_TOKEN`
+   - `CROWDSEC_LAPI_KEY`
+2. Render secret-aware dynamic files:
    ```bash
-    git clone https://gitea.gbanyan.net/gbanyan/GB-Traefik.git
-    cd GB-Traefik
+   ./scripts/render_dynamic.sh
    ```
+   This uses `templates/crowdsec.yml.tmpl` and writes `dynamic.d/middlewares/crowdsec.yml` (ignored by git).
 
-2. Update the `traefik.yml` and `docker-compose.yml` files as needed for your environment.
+## Runbook
 
-3. Start Traefik:
 ```bash
-    docker compose up -d
+# start / update Traefik
+docker compose up -d traefik
+
+# refresh Cloudflare IPs and restart safely
+python scripts/update_cloudflare_ips.py
+
+# tail logs
+tail -f traefik.log
+tail -f access.log
 ```
 
-4. Access the Traefik dashboard (if enabled) at `http://<your-domain-or-ip>:8080`.
+Rotate CrowdSec keys? Edit `.env`, rerun `render_dynamic.sh`, then `docker compose up -d traefik`.
 
-## Configuration
-
-- **.env**: Cloudflare E-mail and API Token for SSL DNS Challenge
-- **Traefik Configuration**: Modify `traefik.yml`, `dynamic.yml` to customize Traefik's behavior.
-- **Docker Compose**: Use `docker-compose.yml` to define services and networks.
-
-
-## Detail: 
-
-My traefik is split into internal and external entrypoint. 
-
-Internal entrypoint is for private and secure service without exposing. 
-
-Each entrypoint is binded to different ip address for isolation. 
-
-Then, other docker service is attached to different entrypoint guided by label in docker compose
-
-```yaml
-label: 
-    - "traefik.http.routers.service-name.entrypoints=websecure"
-```
-
-Besides the entrypoint setup, I add CrowdSec firewall bouncer plus a compression middleware (brotli/gzip/zstd) defined in `dynamic.yml`. Cloudflare’s IP ranges are injected directly into `traefik.yml` by a helper script, so no extra plugin middleware is required anymore.
-
-Adding middlewares is also guided by labels: 
-
-```yaml
-label: 
-    - "traefik.http.routers.service-name.middlewares=crowdsec@file,compress-middleware@file"
-```
-
-The order of middlewares is meaningful. 
-
-Traefik has ability to apply SSL certs automatically. 
-Just offer the required DNS API authentication (Like cloudflare). 
-
-Please refer the traefik documentation. 
-
-The following is an example of a docker service I hosted in its docker-compose.yaml: 
+## Service Labels Cheat Sheet
 
 ```yaml
 labels:
-      - "traefik.enable=true"
-      - "traefik.http.routers.ghost.entrypoints=websecure"
-      - "traefik.http.routers.ghost.rule=Host(`blog.gbanyan.net`)"
-      - "traefik.http.services.ghost.loadbalancer.server.port=2368"
-      - "traefik.http.routers.ghost.tls.certresolver=letsencrypt"
-      - "traefik.http.routers.ghost.middlewares=crowdsec@file,compress-middleware@file"
-      - "com.centurylinklabs.watchtower.enable=true"
-      - "traefik.docker.network=traefik_default"
+  - traefik.enable=true
+  - traefik.http.routers.myapp.rule=Host(`app.example.com`)
+  - traefik.http.routers.myapp.entrypoints=websecure
+  - traefik.http.routers.myapp.tls.certresolver=letsencrypt
+  - traefik.http.routers.myapp.middlewares=crowdsec@file,retry-fast@file,compress-middleware@file
+  - traefik.http.services.myapp.loadbalancer.serversTransport=fast-upstreams@file
+  - traefik.http.services.myapp.loadbalancer.server.port=3000
+  - traefik.docker.network=traefik_default
 ```
 
-I mount the access.log for crowdsec firewall to read. 
+Most stacks use the same middleware chain: CrowdSec bouncer (plugin), retry, and compression. Internal-only services skip CrowdSec by pointing at the `internal_*` entrypoints.
 
-PS: Because I access my traefik dashboard through my local network. I commented out the authetication method for dashboard. 
+## Performance Notes
 
-## Discussion and Changelog
+- Access logs are buffered with headers trimmed to keep syscalls down.  
+- Compression enforces a 1 KB minimum and respects the client’s preferred encoding.  
+- Shared transport keeps 64 idle connections per backend with aggressive idle/response timeouts.  
+- `retry-fast` retries once after 50 ms, smoothing transient Puma/Node hiccups without hammering backends.
 
-1. Traefik vs Nginx
-- Performance: Nginx is still better at high traffic. After all it is written in C. Traefik 3 though claims it has higher 20% performance than before. The latency still showed a little higher than nginx. 
-- Docker Deployment Ease: Traefik is easier for docker service deployment. In my environment, I can assign each docker stack with labels and then guides the traefik to add Let's encrypt SSL. 
+## Things to Remember
 
-2. ChangeLog: 
-
-- 2025.4.21 Add the defaulthost rule for container name for lazy writing. But commented out for precision.  
-- 2025.4.21 Fix the trusted IP settings; later replaced by an internal updater instead of the traefik-plugin-cloudflare. 
-- 2025.4.18 Add Souin HTTP Cache Middleware (in feature branch, not merge into main)
-- 2025.4.18 Temp disable the compression middleware. It has MIME type bugs. 
-
-## Notes on Host Networking
-
-Traefik currently runs with `network_mode: host` so it can bind directly to both `10.0.0.225` (public) and `192.168.50.4` (internal) entrypoints. Moving back to bridge mode would break that dual-IP isolation because Docker cannot publish the same container port on two different host interfaces. Host networking also means:
-
-- Traefik reaches app containers like any other host process, ignoring `traefik.docker.network` labels.
-- Linux handles firewalling/routing between the two interfaces; Docker’s conntrack optimizations aren’t used.
-
-If you ever want to switch to bridge networking, you’d need either separate Traefik instances (one per subnet) or an external L4 proxy in front of a single Traefik that listens on generic `:80/:443` ports. For now the host-mode trade-off is intentional to keep the internal/external split simple.
+- Watchtower is still enabled for Traefik; pin the image tag when you need deterministic upgrades.  
+- `scripts/update_cloudflare_ips.py` rewrites the static trusted IP block and restarts Traefik—run it via cron.  
+- Dashboard auth is intentionally disabled because access only happens from the LAN entrypoint. If that changes, re-enable `basicauth`.

docker-compose.yaml

@@ -17,7 +17,7 @@ services:
       - /var/run/docker.sock:/var/run/docker.sock:ro
       - ./certs:/letsencrypt
       #- ./dashboard_authfile:/dashboard_authfile:ro
-      - ./dynamic.yml:/dynamic.yml
+      - ./dynamic.d:/dynamic.d
       - ./traefik.yml:/traefik.yml
       - ./traefik.log:/var/log/traefik/traefik.log
       - ./access.log:/var/log/traefik/access.log
@@ -47,4 +47,3 @@ networks:
       config:
         - subnet: 172.19.0.0/16
           gateway: 172.19.0.1
-

dynamic.d/middlewares/compress.yml

@@ -0,0 +1,15 @@
+http:
+  middlewares:
+    compress-middleware:
+      compress:
+        encodings:
+          - zstd
+          - br
+          - gzip
+        minResponseBodyBytes: 1024
+        includedContentTypes:
+          - text/html
+          - text/css
+          - application/javascript
+          - application/json
+          - text/plain

dynamic.d/middlewares/retry.yml

@@ -0,0 +1,6 @@
+http:
+  middlewares:
+    retry-fast:
+      retry:
+        attempts: 2
+        initialInterval: 50ms

dynamic.d/routers/internal.yml

@@ -0,0 +1,14 @@
+http:
+  routers:
+    qbit:
+      rule: Host(`qbit.gbanyan.net`)
+      service: qbit
+      entryPoints:
+        - internal_websecure
+      tls:
+        certResolver: letsencrypt
+  services:
+    qbit:
+      loadBalancer:
+        servers:
+          - url: "http://192.168.50.4:8083"

dynamic.d/transports/fast-upstreams.yml

@@ -0,0 +1,7 @@
+http:
+  serversTransports:
+    fast-upstreams:
+      maxIdleConnsPerHost: 64
+      forwardingTimeouts:
+        idleConnTimeout: 30s
+        responseHeaderTimeout: 15s

dynamic.yml

@@ -1,35 +0,0 @@
-http:
-  middlewares:
-    crowdsec:
-      plugin:
-        bouncer:
-          enabled: true
-          crowdsecMode: stream
-          crowdsecLapiHost: "localhost:8080"
-          crowdsecLapiKey: gFJjSzdbB0GCe/1Y9HcxMPP1vQmoa4psZOFyleJZJVQ
-    compress-middleware:
-      compress:
-        encodings:
-          - zstd
-          - br
-          - gzip
-        defaultEncoding: zstd
-        includedContentTypes:
-          - text/html
-          - text/css
-          - application/javascript
-          - application/json
-          - text/plain
-  routers:
-    qbit:
-      rule: Host(`qbit.gbanyan.net`)
-      service: qbit
-      entryPoints: ["internal_websecure"]
-      tls:
-        certResolver: letsencrypt
-
-  services:
-    qbit:
-      loadBalancer:
-        servers:
-          - url: "http://192.168.50.4:8083"

scripts/render_dynamic.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+set -euo pipefail
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ROOT_DIR"
+if [[ ! -f .env ]]; then
+  echo "Missing .env file. Copy .env.example and fill in secrets." >&2
+  exit 1
+fi
+set -a
+# shellcheck disable=SC1091
+source .env
+set +a
+: "${CROWDSEC_LAPI_KEY:?CROWDSEC_LAPI_KEY must be set in .env}"
+if ! command -v envsubst >/dev/null 2>&1; then
+  echo "envsubst is required to render templates." >&2
+  exit 1
+fi
+envsubst < templates/crowdsec.yml.tmpl > dynamic.d/middlewares/crowdsec.yml
+echo "Rendered dynamic.d/middlewares/crowdsec.yml"

templates/crowdsec.yml.tmpl

@@ -0,0 +1,9 @@
+http:
+  middlewares:
+    crowdsec:
+      plugin:
+        bouncer:
+          enabled: true
+          crowdsecMode: stream
+          crowdsecLapiHost: "localhost:8080"
+          crowdsecLapiKey: "${CROWDSEC_LAPI_KEY}"

traefik.yml

@@ -85,7 +85,6 @@ entryPoints:
         - "2405:8100::/32"
         - "2a06:98c0::/29"
         - "2c0f:f248::/32"
-    http3: {}
   internal_web:
     address: "192.168.50.4:80"
     http:
@@ -95,7 +94,6 @@ entryPoints:
             scheme: "https"
   internal_websecure:
     address: "192.168.50.4:443"
-    http3: {}
   metrics:
     address: ":8082"
   dashboard:
@@ -110,7 +108,7 @@ providers:
     exposedByDefault: false
     # defaultRule: "Host(`{{ .ContainerName }}.gbanyan.net`)"
   file:
-    filename: "/dynamic.yml"  # Enable dynamic configuration file
+    directory: "/dynamic.d"
 certificatesResolvers:
   letsencrypt:
     acme: