OK here we go:
I did loosely based on this guide: https://docs.goauthentik.io/integrations/services/home-assistant/

1- Install HACS add-on: https://github.com/BeryJu/hass-auth-header (Click on add repository and this link) 2- Create the Authentik instance and the outpost:

Type: proxy
Advanced settings:
authentik_host: http://authentik.yourdns.com

3- Create a provider:

Name: HomeAssistant
Type: Proxy - Forward Auth(domain level)
Authentication flow: default-authentication-flow
Authorization flow: default-provider-authorization-explicit-consent
Authentication URL: https://homeassistant.yourdns.com
Cookie domain: yourdns.com

4- Create a application: Just create a new HomeAssistant application with your information and click create. 5- Attach application to the Provider and then Provider to the Outpost.

6- Now that you have the integration for Auth Headers create this at configuration.yaml and restart HomeAssistant

OBS: Remember to edit all the upstream and proxy-passes with your IP/Hostname/Port, etc of your instances.

HomeAssistant configuration.yaml:

# Reverse proxy configuration
http:
    use_x_forwarded_for: true
    trusted_proxies:
        - x.x.x.x/32 # This needs to be set to the IP of your reverse proxy and subnet size that you want to allow

auth_header:
    username_header: X-authentik-username
    debug: true
# Optionally, if something is not working right, add this block below to get more information
logger:
    default: info
    logs:
        custom_components.auth_header: debug

I'm using SWAG for the reverse proxy, but nginx should work as well.

Just follow the "include" files and you should be fine.

authentik.server.conf

location ^~ /outpost.goauthentik.io {
    auth_request off; # requests to this subfolder must be accessible without authentication
    include /config/nginx/proxy.conf;
    include /config/nginx/resolver.conf;
    set $upstream_authentik authentik-server;
    proxy_pass http://$upstream_authentik:9000;
}

# location for authentik auth requests
location = /outpost.goauthentik.io/auth/nginx {
    internal;

    include /config/nginx/proxy.conf;
    include /config/nginx/resolver.conf;
    set $upstream_authentik authentik-server;
    proxy_pass http://$upstream_authentik:9000;

    ## Include the Set-Cookie header if present
    auth_request_set $set_cookie $upstream_http_set_cookie;
    add_header Set-Cookie $set_cookie;

    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
}

# virtual location for authentik 401 redirects
location @goauthentik_proxy_signin {
    internal;

    ## Include the Set-Cookie header if present
    auth_request_set $set_cookie $upstream_http_set_cookie;
    add_header Set-Cookie $set_cookie;

    ## Set the $target_url variable based on the original request
    set_escape_uri $target_url $scheme://$http_host$request_uri;

    ## Set the $signin_url variable
    set $signin_url https://$http_host/outpost.goauthentik.io/start?rd=$target_url;

    ## Redirect to login
    return 302 $signin_url;
}

authentik-location.conf

## Send a subrequest to Authentik to verify if the user is authenticated and has permission to access the resource
auth_request /outpost.goauthentik.io/auth/nginx;
## If the subreqest returns 200 pass to the backend, if the subrequest returns 401 redirect to the portal
error_page 401 = @goauthentik_proxy_signin;

## Translate the user information response headers from the auth subrequest into variables
auth_request_set $authentik_email $upstream_http_x_authentik_email;
auth_request_set $authentik_groups $upstream_http_x_authentik_groups;
auth_request_set $authentik_name $upstream_http_x_authentik_name;
auth_request_set $authentik_uid $upstream_http_x_authentik_uid;
auth_request_set $authentik_username $upstream_http_x_authentik_username;

## Inject the user information into the request made to the actual upstream
proxy_set_header X-authentik-email $authentik_email;
proxy_set_header X-authentik-groups $authentik_groups;
proxy_set_header X-authentik-name $authentik_name;
proxy_set_header X-authentik-uid $authentik_uid;
proxy_set_header X-authentik-username $authentik_username;

## Translate the Set-Cookie response header from the auth subrequest into a variable
auth_request_set $set_cookie $upstream_http_set_cookie;

Now the proxy-conf for the authentik first, so it goes to the reverse-proxy to create the authentik.yourdns.com

## Version 2023/05/31
# make sure that your authentik container is named authentik-server
# make sure that your dns has a cname set for authentik

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name authentik.*;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    location / {

        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app authentik-server;
        set $upstream_port 9000;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location ~ (/authentik)?/api {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app authentik-server;
        set $upstream_port 9000;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }
}

Then the homeassistant configuration to enable the authentik on the subdomain: (remember to follow the includes from above)

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name homeassistant.*;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    # enable for Authentik (requires authentik-location.conf in the location block)
    include /config/nginx/authentik-server.conf;

    location / {
        # enable for Authentik (requires authentik-server.conf in the server block)
        include /config/nginx/authentik-location.conf;

        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app homeassistant;
        set $upstream_port 8123;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location ~ ^/(api|local|media)/ {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app homeassistant;
        set $upstream_port 8123;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;
    }
}

With this I managed to make it work, even in android app its working now.

Troubleshooting: Go check the logs of homeassistant, that debug will help identify the issue: If it says: no callback something it's the provider of Authentik that's wrong, if it says "No header set, returning empty flow" and "no matching user found" create a new user at HomeAssistant and the same username as admin on the Authentik, this was my error before, once I created a new proper user on both sides it started to work.