Increase expiry of tokens.

No need to account for revisions that are not current when calculating stokens
because those, by definition, are not the latest ones, and therefore won't have
the most recent stokens.

This becomes a problem when collections have many associated revisions.

.dockerignore

@@ -0,0 +1,16 @@
+/db.sqlite3*
+Session.vim
+/local_settings.py
+.venv
+/assets
+/logs
+/.coverage
+/tmp
+/media
+
+__pycache__
+.*.swp
+
+/.*
+
+/sandbox

.gitignore

@@ -14,3 +14,7 @@ __pycache__
 
 /etebase_server_settings.py
 /secret.txt
+
+/build
+/dist
+/*.egg-info

ChangeLog.md

@@ -1,5 +1,41 @@
 # Changelog
 
+## Version 0.11.0
+- Update deps for Python 3.11
+
+## Version 0.10.0
+- Replace the deprecated aioredis with redis-py
+- Optimize how we fetch the current (latest) revision of items
+
+## Version 0.9.1
+- Update pinned Django version (only matters if using `requirements.txt`).
+
+## Version 0.9.0
+- Add LDAP support for checking the validity of a username
+- Allow specifying engine-specific database options
+- Fix crash on shutdown when redis isn't used
+- Reorganize the code to be a valid Python package
+
+## Version 0.8.3
+- Fix compatibility with latest fastapi
+
+## Version 0.8.2
+- Update dependencies again
+
+## Version 0.8.1
+* Fix Error `404 Not Found` for Static Files
+* Fix Django 3.2 warnings
+* Update dependencies while (keep Django 3.2 LTS)
+
+## Version 0.8.0
+* Update django dep.
+* Fix issue with comparing ports in hostname verification with self-hosted servers.
+* Fix sendfile settings to be more correct.
+* Improve easy config (make it clear media_root needs to be set)
+* Handle stoken being the empty string
+* Fix mysql/mariadb support
+* Switch to FastAPI for the server component
+
 ## Version 0.7.0
 * Chunks: improve the chunk download endpoint to use sendfile extensions
 * Chunks: support not passing chunk content if exists

README.md

@@ -63,6 +63,12 @@ Now you can initialise our django app.
 ./manage.py migrate
 ```
 
+Create static files:
+
+```
+./manage.py collectstatic
+```
+
 And you are done! You can now run the debug server just to see everything works as expected by running:
 
 ```
@@ -109,9 +115,10 @@ The default configuration creates a file “`secret.txt`” in the project’s
 base directory, which is used as the value of the Django `SECRET_KEY`
 setting. You can revoke this key by deleting the `secret.txt` file and the
 next time the app is run, a new one will be generated. Make sure you keep
-the `secret.txt` file secret (don’t accidentally commit it to version
-control, exclude it from your backups, etc.). If you want to change to a
-more secure system for storing secrets, edit `etesync_server/settings.py`
+the `secret.txt` file secret (e.g. don’t accidentally commit it to version
+control). However, backing it up is okay, and it makes it easier to restore
+the database to a new EteSync server, but it's not essential. If you want to
+change to a more secure system for storing secrets, edit `etesync_server/settings.py`
 and implement your own method for setting `SECRET_KEY` (remove the line
 where it uses the `get_secret_from_file` function).  Read the Django docs
 for more information about the `SECRET_KEY` and its uses.
@@ -138,6 +145,20 @@ Here are the update steps:
 4. Run the migration tool to migrate all of your data.
 5. Add your new EteSync 2.0 accounts to all of your devices.
 
+# Testing
+
+Docker images named `etesync/test-server:<version>` and `:latest` are available for testing etesync clients.
+This docker image starts a server on port 3735 that supports user signup (without email confirmation), is in debug mode (thus supporting the reset endpoint), and stores its data locally.
+It is in no way suitable for production usage, but is able to start up quickly and makes a good component of CI for etesync clients and users of those clients.
+
+# User signup
+
+Instead of having to create Django users manually when signup up Etebase users, it is also possible to allow automatic signup.
+For example, this makes sense when putting an Etebase server in production.
+However, this does come with the added risk that everybody with access to your server will be able to sign up.
+
+In order to set it up, comment out the line `ETEBASE_CREATE_USER_FUNC = "etebase_server.django.utils.create_user_blocked"` in `server/settings.py` and restart your Etebase server.
+
 # License
 
 Etebase is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License version 3 as published by the Free Software Foundation. See the [LICENSE](./LICENSE) for more information.
@@ -154,7 +175,13 @@ Please consider registering an account even if you self-host in order to support
 
 Become a financial contributor and help us sustain our community!
 
+## Supporters ($20 / month)
+
+[![jzacsh](https://github.com/jzacsh.png?size=80)](https://github.com/jzacsh)
+
 ## Contributors ($10 / month)
 
 [![ilovept](https://github.com/ilovept.png?size=40)](https://github.com/ilovept)
 [![ryanleesipes](https://github.com/ryanleesipes.png?size=40)](https://github.com/ryanleesipes)
+[![DanielG](https://github.com/DanielG.png?size=40)](https://github.com/DanielG)
+[![Kanaye](https://github.com/Kanaye.png?size=40)](https://github.com/Kanaye)

django_etebase/signals.py

@@ -1,3 +0,0 @@
-from django.dispatch import Signal
-
-user_signed_up = Signal(providing_args=["request", "user"])

docker/build.sh

@@ -0,0 +1,16 @@
+#! /bin/bash
+
+# Build the `test-server` image, which runs the server in a simple configuration
+# designed to be used in tests, based on the current git revision.
+
+TAG="${1:-latest}"
+
+echo "Building working copy to etesync/test-server:${TAG}"
+
+ETESYNC_VERSION=$(git describe --tags)
+
+docker build \
+    --build-arg ETESYNC_VERSION=${ETESYNC_VERSION} \
+    -t etesync/test-server:${TAG} \
+    -f docker/test-server/Dockerfile \
+    .

docker/test-server/Dockerfile

@@ -0,0 +1,38 @@
+FROM python:3.11.0-alpine
+
+ARG ETESYNC_VERSION
+
+ENV PIP_DISABLE_PIP_VERSION_CHECK=1
+ENV PIP_NO_CACHE_DIR=1
+
+# install packages and pip requirements first, in a single step,
+COPY /requirements.txt /requirements.txt
+RUN set -ex ;\
+    apk add libpq postgresql-dev --virtual .build-deps coreutils gcc libc-dev libffi-dev make ;\
+    pip install -U pip ;\
+    pip install --no-cache-dir --progress-bar off -r /requirements.txt ;\
+    apk del .build-deps make gcc coreutils ;\
+    rm -rf /root/.cache
+
+COPY . /app
+
+RUN set -ex ;\
+    mkdir -p /data/static /data/media ;\
+    cd /app ;\
+    mkdir -p /etc/etebase-server ;\
+    cp docker/test-server/etebase-server.ini /etc/etebase-server ;\
+    sed -e '/ETEBASE_CREATE_USER_FUNC/ s/^#*/#/' -i /app/etebase_server/settings.py ;\
+    chmod +x docker/test-server/entrypoint.sh
+
+# this is a test image and should start up quickly, so it starts with the DB
+# and static data already fully set up.
+RUN set -ex ;\
+    cd /app ;\
+    python manage.py migrate ;\
+    python manage.py collectstatic --noinput
+
+ENV ETESYNC_VERSION=${ETESYNC_VERSION}
+VOLUME /data
+EXPOSE 3735
+
+ENTRYPOINT ["/app/docker/test-server/entrypoint.sh"]

docker/test-server/entrypoint.sh

@@ -0,0 +1,6 @@
+#! /bin/sh
+
+echo "Running etesync test server ${ETESYNC_VERSION}"
+
+cd /app
+uvicorn etebase_server.asgi:application --host 0.0.0.0 --port 3735

docker/test-server/etebase-server.ini

@@ -0,0 +1,12 @@
+[global]
+secret_file = secret.txt
+debug = true
+static_root = /data/static
+media_root = /data/media
+
+[allowed_hosts]
+allowed_host1 = *
+
+[database]
+engine = django.db.backends.sqlite3
+name = /data/db.sqlite3

etebase-server.ini.example

@@ -1,10 +1,12 @@
 [global]
 secret_file = secret.txt
 debug = false
+;Set the paths where data will be stored at
+static_root = /path/to/static
+media_root = /path/to/media
+
 ;Advanced options, only uncomment if you know what you're doing:
-;static_root = /path/to/static
 ;static_url = /static/
-;media_root = /path/to/media
 ;media_url = /user-media/
 ;language_code = en-us
 ;time_zone = UTC
@@ -16,3 +18,18 @@ allowed_host1 = example.com
 [database]
 engine = django.db.backends.sqlite3
 name = db.sqlite3
+
+[database-options]
+; Add engine-specific options here, such as postgresql parameter key words
+
+;[ldap]
+;server = <The URL to your LDAP server>
+;search_base = <Your search base>
+;filter = <Your LDAP filter query. '%%s' will be substituted for the username>
+; In case a cache TTL of 1 hour is too short for you, set `cache_ttl` to the preferred
+; amount of hours a cache entry should be viewed as valid:
+;cache_ttl = 5
+;bind_dn = <Your LDAP "user" to bind as. Must be a bind user>
+; Either specify the password directly, or provide a password file
+;bind_pw = <The password to authenticate as your bind user>
+;bind_pw_file = /path/to/the/file.txt

etebase_server/asgi.py

@@ -7,7 +7,7 @@ django_application = get_asgi_application()
 
 
 def create_application():
-    from etebase_fastapi.main import create_application
+    from etebase_server.fastapi.main import create_application
 
     app = create_application()
 

etebase_server/django/apps.py

@@ -2,4 +2,5 @@ from django.apps import AppConfig
 
 
 class DjangoEtebaseConfig(AppConfig):
-    name = "django_etebase"
+    name = "etebase_server.django"
+    label = "django_etebase"

etebase_server/django/migrations/0001_initial.py

@@ -4,7 +4,7 @@ from django.conf import settings
 import django.core.validators
 from django.db import migrations, models
 import django.db.models.deletion
-import django_etebase.models
+from etebase_server.django.models import chunk_directory_path
 
 
 class Migration(migrations.Migration):
@@ -85,7 +85,7 @@ class Migration(migrations.Migration):
                 ),
                 (
                     "chunkFile",
-                    models.FileField(max_length=150, unique=True, upload_to=django_etebase.models.chunk_directory_path),
+                    models.FileField(max_length=150, unique=True, upload_to=chunk_directory_path),
                 ),
                 (
                     "item",

etebase_server/django/migrations/0008_auto_20200526_1535.py

@@ -3,7 +3,7 @@
 import django.core.validators
 from django.db import migrations, models
 import django.db.models.deletion
-import django_etebase.models
+from etebase_server.django.models import generate_stoken_uid
 
 
 class Migration(migrations.Migration):
@@ -21,7 +21,7 @@ class Migration(migrations.Migration):
                     "uid",
                     models.CharField(
                         db_index=True,
-                        default=django_etebase.models.generate_stoken_uid,
+                        default=generate_stoken_uid,
                         max_length=43,
                         unique=True,
                         validators=[

etebase_server/django/migrations/0021_auto_20200626_0913.py

@@ -2,7 +2,7 @@
 
 import django.core.validators
 from django.db import migrations, models
-import django_etebase.models
+from etebase_server.django.models import generate_stoken_uid
 
 
 class Migration(migrations.Migration):
@@ -62,7 +62,7 @@ class Migration(migrations.Migration):
             name="uid",
             field=models.CharField(
                 db_index=True,
-                default=django_etebase.models.generate_stoken_uid,
+                default=generate_stoken_uid,
                 max_length=43,
                 unique=True,
                 validators=[

etebase_server/django/migrations/0028_auto_20200907_0754.py

@@ -2,7 +2,7 @@
 
 from django.db import migrations
 
-from django_etebase.models import AccessLevels
+from etebase_server.django.models import AccessLevels
 
 
 def change_access_level_to_int(apps, schema_editor):

etebase_server/django/models.py

@@ -66,7 +66,7 @@ class Collection(models.Model):
     @cached_property
     def stoken(self) -> str:
         stoken_id = (
-            self.__class__.objects.filter(main_item=self.main_item)
+            self.__class__.objects.filter(main_item=self.main_item, items__revisions__current=True)
             .annotate(max_stoken=self.stoken_annotation)
             .values("max_stoken")
             .first()["max_stoken"]
@@ -96,7 +96,7 @@ class CollectionItem(models.Model):
 
     @cached_property
     def content(self) -> "CollectionItemRevision":
-        return self.revisions.get(current=True)
+        return self.revisions.filter(current=True)[0]
 
     @property
     def etag(self) -> str:

etebase_server/django/signals.py

@@ -0,0 +1,4 @@
+from django.dispatch import Signal
+
+# Provides arguments "request" and "user"
+user_signed_up = Signal()

etebase_server/django/token_auth/apps.py

@@ -2,4 +2,4 @@ from django.apps import AppConfig
 
 
 class TokenAuthConfig(AppConfig):
-    name = "django_etebase.token_auth"
+    name = "etebase_server.django.token_auth"

etebase_server/django/token_auth/migrations/0001_initial.py

@@ -3,7 +3,7 @@
 from django.conf import settings
 from django.db import migrations, models
 import django.db.models.deletion
-from django_etebase.token_auth import models as token_auth_models
+from etebase_server.django.token_auth import models as token_auth_models
 
 
 class Migration(migrations.Migration):

etebase_server/django/token_auth/models.py

@@ -1,7 +1,7 @@
 from django.db import models
 from django.utils import timezone
 from django.utils.crypto import get_random_string
-from myauth.models import get_typed_user_model
+from etebase_server.myauth.models import get_typed_user_model
 
 User = get_typed_user_model()
 
@@ -11,7 +11,7 @@ def generate_key():
 
 
 def get_default_expiry():
-    return timezone.now() + timezone.timedelta(days=30)
+    return timezone.now() + timezone.timedelta(days=365 * 20)
 
 
 class AuthToken(models.Model):

etebase_server/django/utils.py

@@ -3,7 +3,7 @@ from dataclasses import dataclass
 
 from django.db.models import QuerySet
 from django.core.exceptions import PermissionDenied
-from myauth.models import UserType, get_typed_user_model
+from etebase_server.myauth.models import UserType, get_typed_user_model
 
 from . import app_settings
 

etebase_server/fastapi/dependencies.py

@@ -6,9 +6,9 @@ from fastapi.security import APIKeyHeader
 from django.utils import timezone
 from django.db.models import QuerySet
 
-from django_etebase import models
-from django_etebase.token_auth.models import AuthToken, get_default_expiry
-from myauth.models import UserType, get_typed_user_model
+from etebase_server.django import models
+from etebase_server.django.token_auth.models import AuthToken, get_default_expiry
+from etebase_server.myauth.models import UserType, get_typed_user_model
 from .exceptions import AuthenticationFailed
 from .utils import get_object_or_404
 from .db_hack import django_db_cleanup_decorator

etebase_server/fastapi/main.py

@@ -4,8 +4,9 @@ from django.conf import settings
 from fastapi import FastAPI, Request
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.middleware.trustedhost import TrustedHostMiddleware
+from fastapi.staticfiles import StaticFiles
 
-from django_etebase import app_settings
+from etebase_server.django import app_settings
 
 from .exceptions import CustomHttpException
 from .msgpack import MsgpackResponse
@@ -42,7 +43,7 @@ def create_application(prefix="", middlewares=[]):
     app.include_router(websocket_router, prefix=f"{BASE_PATH}/ws", tags=["websocket"])
 
     if settings.DEBUG:
-        from etebase_fastapi.routers.test_reset_view import test_reset_view_router
+        from .routers.test_reset_view import test_reset_view_router
 
         app.include_router(test_reset_view_router, prefix=f"{BASE_PATH}/test/authentication")
 
@@ -74,4 +75,6 @@ def create_application(prefix="", middlewares=[]):
     async def custom_exception_handler(request: Request, exc: CustomHttpException):
         return MsgpackResponse(status_code=exc.status_code, content=exc.as_dict)
 
+    app.mount(settings.STATIC_URL, StaticFiles(directory=settings.STATIC_ROOT), name="static")
+
     return app

etebase_server/fastapi/msgpack.py

@@ -12,7 +12,7 @@ from .db_hack import django_db_cleanup_decorator
 class MsgpackRequest(Request):
     media_type = "application/msgpack"
 
-    async def json(self) -> bytes:
+    async def body(self) -> bytes:
         if not hasattr(self, "_json"):
             body = await super().body()
             self._json = msgpack_decode(body)

etebase_server/fastapi/redis.py

@@ -1,7 +1,7 @@
 import typing as t
-import aioredis
+from redis import asyncio as aioredis
 
-from django_etebase import app_settings
+from etebase_server.django import app_settings
 
 
 class RedisWrapper:
@@ -12,12 +12,11 @@ class RedisWrapper:
 
     async def setup(self):
         if self.redis_uri is not None:
-            self.redis = await aioredis.create_redis_pool(self.redis_uri)
+            self.redis = await aioredis.from_url(self.redis_uri)
 
     async def close(self):
-        if self.redis is not None:
-            self.redis.close()
-            await self.redis.wait_closed()
+        if hasattr(self, "redis"):
+            await self.redis.close()
 
     @property
     def is_active(self):

etebase_server/fastapi/routers/authentication.py

@@ -14,12 +14,12 @@ from django.db import transaction
 from django.utils.functional import cached_property
 from fastapi import APIRouter, Depends, status, Request
 
-from django_etebase import app_settings, models
-from django_etebase.token_auth.models import AuthToken
-from django_etebase.models import UserInfo
-from django_etebase.signals import user_signed_up
-from django_etebase.utils import create_user, get_user_queryset, CallbackContext
-from myauth.models import UserType, get_typed_user_model
+from etebase_server.django import app_settings, models
+from etebase_server.django.token_auth.models import AuthToken
+from etebase_server.django.models import UserInfo
+from etebase_server.django.signals import user_signed_up
+from etebase_server.django.utils import create_user, get_user_queryset, CallbackContext
+from etebase_server.myauth.models import UserType, get_typed_user_model
 from ..exceptions import AuthenticationFailed, transform_validation_error, HttpError
 from ..msgpack import MsgpackRoute
 from ..utils import BaseModel, permission_responses, msgpack_encode, msgpack_decode, get_user_username_email_kwargs
@@ -161,7 +161,7 @@ def validate_login_request(
         raise HttpError("challenge_expired", "Login challenge has expired")
     elif challenge_data["userId"] != user.id:
         raise HttpError("wrong_user", "This challenge is for the wrong user")
-    elif not settings.DEBUG and validated_data.host.split(":", 1)[0] != host_from_request:
+    elif not settings.DEBUG and validated_data.host.split(":", 1)[0] != host_from_request.split(":", 1)[0]:
         raise HttpError(
             "wrong_host", f'Found wrong host name. Got: "{validated_data.host}" expected: "{host_from_request}"'
         )

etebase_server/fastapi/routers/collection.py

@@ -7,8 +7,8 @@ from django.db import transaction, IntegrityError
 from django.db.models import Q, QuerySet
 from fastapi import APIRouter, Depends, status, Request, BackgroundTasks
 
-from django_etebase import models
-from myauth.models import UserType
+from etebase_server.django import models
+from etebase_server.myauth.models import UserType
 from .authentication import get_authenticated_user
 from .websocket import get_ticket, TicketRequest, TicketOut
 from ..exceptions import HttpError, transform_validation_error, PermissionDenied, ValidationError
@@ -208,7 +208,7 @@ def collection_list_common(
     prefetch: Prefetch,
 ) -> CollectionListResponse:
     result, new_stoken_obj, done = filter_by_stoken_and_limit(
-        stoken, limit, queryset, models.Collection.stoken_annotation
+        stoken, limit, queryset.filter(items__revisions__current=True), models.Collection.stoken_annotation
     )
     new_stoken = new_stoken_obj and new_stoken_obj.uid
     context = Context(user, prefetch)
@@ -396,7 +396,7 @@ def item_create(item_model: CollectionItemIn, collection: models.Collection, val
         if not created:
             # We don't have to use select_for_update here because the unique constraint on current guards against
             # the race condition. But it's a good idea because it'll lock and wait rather than fail.
-            current_revision = instance.revisions.filter(current=True).select_for_update().first()
+            current_revision = instance.revisions.filter(current=True).select_for_update()[0]
             assert current_revision is not None
             current_revision.current = None
             current_revision.save()
@@ -428,7 +428,7 @@ def item_list_common(
     prefetch: Prefetch,
 ) -> CollectionItemListResponse:
     result, new_stoken_obj, done = filter_by_stoken_and_limit(
-        stoken, limit, queryset, models.CollectionItem.stoken_annotation
+        stoken, limit, queryset.filter(revisions__current=True), models.CollectionItem.stoken_annotation
     )
     new_stoken = new_stoken_obj and new_stoken_obj.uid
     context = Context(user, prefetch)

etebase_server/fastapi/routers/invitation.py

@@ -4,9 +4,9 @@ from django.db import transaction, IntegrityError
 from django.db.models import QuerySet
 from fastapi import APIRouter, Depends, status, Request
 
-from django_etebase import models
-from django_etebase.utils import get_user_queryset, CallbackContext
-from myauth.models import UserType, get_typed_user_model
+from etebase_server.django import models
+from etebase_server.django.utils import get_user_queryset, CallbackContext
+from etebase_server.myauth.models import UserType, get_typed_user_model
 from .authentication import get_authenticated_user
 from ..exceptions import HttpError, PermissionDenied
 from ..msgpack import MsgpackRoute

etebase_server/fastapi/routers/member.py

@@ -4,8 +4,8 @@ from django.db import transaction
 from django.db.models import QuerySet
 from fastapi import APIRouter, Depends, status
 
-from django_etebase import models
-from myauth.models import UserType, get_typed_user_model
+from etebase_server.django import models
+from etebase_server.myauth.models import UserType, get_typed_user_model
 from .authentication import get_authenticated_user
 from ..msgpack import MsgpackRoute
 from ..utils import get_object_or_404, BaseModel, permission_responses, PERMISSIONS_READ, PERMISSIONS_READWRITE

etebase_server/fastapi/routers/test_reset_view.py

@@ -3,11 +3,11 @@ from django.db import transaction
 from django.shortcuts import get_object_or_404
 from fastapi import APIRouter, Request, status
 
-from django_etebase.utils import get_user_queryset, CallbackContext
+from etebase_server.django.utils import get_user_queryset, CallbackContext
 from .authentication import SignupIn, signup_save
 from ..msgpack import MsgpackRoute
 from ..exceptions import HttpError
-from myauth.models import get_typed_user_model
+from etebase_server.myauth.models import get_typed_user_model
 
 test_reset_view_router = APIRouter(route_class=MsgpackRoute, tags=["test helpers"])
 User = get_typed_user_model()

etebase_server/fastapi/routers/websocket.py

@@ -1,16 +1,17 @@
 import asyncio
 import typing as t
 
-import aioredis
+from redis import asyncio as aioredis
+from redis.exceptions import ConnectionError
 from asgiref.sync import sync_to_async
 from django.db.models import QuerySet
 from fastapi import APIRouter, Depends, WebSocket, WebSocketDisconnect, status
 import nacl.encoding
 import nacl.utils
 
-from django_etebase import models
-from django_etebase.utils import CallbackContext, get_user_queryset
-from myauth.models import UserType, get_typed_user_model
+from etebase_server.django import models
+from etebase_server.django.utils import CallbackContext, get_user_queryset
+from etebase_server.myauth.models import UserType, get_typed_user_model
 
 from ..dependencies import get_collection_queryset, get_item_queryset
 from ..exceptions import NotSupported
@@ -51,7 +52,7 @@ async def get_ticket(
     uid = nacl.encoding.URLSafeBase64Encoder.encode(nacl.utils.random(32))
     ticket_model = TicketInner(user=user.id, req=ticket_request)
     ticket_raw = msgpack_encode(ticket_model.dict())
-    await redisw.redis.set(uid, ticket_raw, expire=TICKET_VALIDITY_SECONDS * 1000)
+    await redisw.redis.set(uid, ticket_raw, ex=TICKET_VALIDITY_SECONDS * 1000)
     return TicketOut(ticket=uid)
 
 
@@ -103,9 +104,9 @@ async def send_item_updates(
 
 async def redis_connector(websocket: WebSocket, ticket_model: TicketInner, user: UserType, stoken: t.Optional[str]):
     async def producer_handler(r: aioredis.Redis, ws: WebSocket):
+        pubsub = r.pubsub()
         channel_name = f"col.{ticket_model.req.collection}"
-        (channel,) = await r.psubscribe(channel_name)
-        assert isinstance(channel, aioredis.Channel)
+        await pubsub.subscribe(channel_name)
 
         # Send missing items if we are not up to date
         queryset: QuerySet[models.Collection] = get_collection_queryset(user)
@@ -117,26 +118,29 @@ async def redis_connector(websocket: WebSocket, ticket_model: TicketInner, user:
             return
         await send_item_updates(websocket, collection, user, stoken)
 
+        async def handle_message():
+            msg = await pubsub.get_message(ignore_subscribe_messages=True, timeout=20)
+            message_raw = t.cast(t.Optional[t.Tuple[str, bytes]], msg)
+            if message_raw:
+                _, message = message_raw
+                await ws.send_bytes(message)
+
         try:
             while True:
                 # We wait on the websocket so we fail if web sockets fail or get data
                 receive = asyncio.create_task(websocket.receive())
                 done, pending = await asyncio.wait(
-                    {receive, channel.wait_message()}, return_when=asyncio.FIRST_COMPLETED
+                    {receive, handle_message()},
+                    return_when=asyncio.FIRST_COMPLETED,
                 )
                 for task in pending:
                     task.cancel()
                 if receive in done:
-                    # Web socket should never receieve any data
+                    # Web socket should never receive any data
                     await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
                     return
 
-                message_raw = t.cast(t.Optional[t.Tuple[str, bytes]], await channel.get())
-                if message_raw:
-                    _, message = message_raw
-                    await ws.send_bytes(message)
-
-        except aioredis.errors.ConnectionClosedError:
+        except ConnectionError:
             await websocket.close(code=status.WS_1012_SERVICE_RESTART)
         except WebSocketDisconnect:
             pass

etebase_server/fastapi/stoken_handler.py

@@ -3,7 +3,7 @@ import typing as t
 from django.db.models import QuerySet
 from fastapi import status
 
-from django_etebase.models import Stoken
+from etebase_server.django.models import Stoken
 
 from .exceptions import HttpError
 

etebase_server/fastapi/utils.py

@@ -10,9 +10,9 @@ from pydantic import BaseModel as PyBaseModel
 from django.db.models import Model, QuerySet
 from django.core.exceptions import ObjectDoesNotExist
 
-from django_etebase import app_settings
-from django_etebase.models import AccessLevels
-from myauth.models import UserType, get_typed_user_model
+from etebase_server.django import app_settings
+from etebase_server.django.models import AccessLevels
+from etebase_server.myauth.models import UserType, get_typed_user_model
 
 from .exceptions import HttpError, HttpErrorOut
 

etebase_server/myauth/apps.py

@@ -2,4 +2,5 @@ from django.apps import AppConfig
 
 
 class MyauthConfig(AppConfig):
-    name = "myauth"
+    name = "etebase_server.myauth"
+    label = "myauth"

etebase_server/myauth/forms.py

@@ -1,6 +1,6 @@
 from django import forms
 from django.contrib.auth.forms import UsernameField
-from myauth.models import get_typed_user_model
+from etebase_server.myauth.models import get_typed_user_model
 
 User = get_typed_user_model()
 

etebase_server/myauth/ldap.py

@@ -0,0 +1,109 @@
+import logging
+
+from django.utils import timezone
+from django.conf import settings
+from django.core.exceptions import PermissionDenied as DjangoPermissionDenied
+from etebase_server.django.utils import CallbackContext
+from etebase_server.myauth.models import get_typed_user_model, UserType
+from etebase_server.fastapi.dependencies import get_authenticated_user
+from etebase_server.fastapi.exceptions import PermissionDenied as FastAPIPermissionDenied
+from fastapi import Depends
+
+import ldap
+
+User = get_typed_user_model()
+
+
+def ldap_setting(name, default):
+    """Wrapper around django.conf.settings"""
+    return getattr(settings, f"LDAP_{name}", default)
+
+
+class LDAPConnection:
+    __instance__ = None
+    __user_cache = {}  # Username -> Valid until
+
+    @staticmethod
+    def get_instance():
+        """To get a Singleton"""
+        if not LDAPConnection.__instance__:
+            return LDAPConnection()
+        else:
+            return LDAPConnection.__instance__
+
+    def __init__(self):
+        # Cache some settings
+        self.__LDAP_FILTER = ldap_setting("FILTER", "")
+        self.__LDAP_SEARCH_BASE = ldap_setting("SEARCH_BASE", "")
+
+        # The time a cache entry is valid (in hours)
+        try:
+            self.__LDAP_CACHE_TTL = int(ldap_setting("CACHE_TTL", ""))
+        except ValueError:
+            logging.error("Invalid value for cache_ttl. Defaulting to 1 hour")
+            self.__LDAP_CACHE_TTL = 1
+
+        password = ldap_setting("BIND_PW", "")
+        if not password:
+            pw_file = ldap_setting("BIND_PW_FILE", "")
+            if pw_file:
+                with open(pw_file, "r") as f:
+                    password = f.read().replace("\n", "")
+
+        self.__ldap_connection = ldap.initialize(ldap_setting("SERVER", ""))
+        try:
+            self.__ldap_connection.simple_bind_s(ldap_setting("BIND_DN", ""), password)
+        except ldap.LDAPError as err:
+            logging.error(f"LDAP Error occurring during bind: {err.desc}")
+
+    def __is_cache_valid(self, username):
+        """Returns True if the cache entry is still valid. Returns False otherwise."""
+        if username in self.__user_cache:
+            if timezone.now() <= self.__user_cache[username]:
+                # Cache entry is still valid
+                return True
+        return False
+
+    def __remove_cache(self, username):
+        del self.__user_cache[username]
+
+    def has_user(self, username):
+        """
+        Since we don't care about the password and so authentication
+        another way, all we care about is whether the user exists.
+        """
+        if self.__is_cache_valid(username):
+            return True
+        if username in self.__user_cache:
+            self.__remove_cache(username)
+
+        filterstr = self.__LDAP_FILTER.replace("%s", username)
+        try:
+            result = self.__ldap_connection.search_s(self.__LDAP_SEARCH_BASE, ldap.SCOPE_SUBTREE, filterstr=filterstr)
+        except ldap.NO_RESULTS_RETURNED:
+            # We handle the specific error first and the the generic error, as
+            # we may expect ldap.NO_RESULTS_RETURNED, but not any other error
+            return False
+        except ldap.LDAPError as err:
+            logging.error(f"Error occurred while performing an LDAP query: {err.desc}")
+            return False
+
+        if len(result) == 1:
+            self.__user_cache[username] = timezone.now() + timezone.timedelta(hours=self.__LDAP_CACHE_TTL)
+            return True
+        return False
+
+
+def is_user_in_ldap(user: UserType = Depends(get_authenticated_user)):
+    if not LDAPConnection.get_instance().has_user(user.username):
+        raise FastAPIPermissionDenied(detail="User not in LDAP directory.")
+
+
+def create_user(context: CallbackContext, *args, **kwargs):
+    """
+    A create_user function which first checks if the user already exists in the
+    configured LDAP directory.
+    """
+    if not LDAPConnection.get_instance().has_user(kwargs["username"]):
+        raise DjangoPermissionDenied("User not in the LDAP directory.")
+    return User.objects.create_user(*args, **kwargs)

etebase_server/myauth/migrations/0002_auto_20200515_0801.py

@@ -1,7 +1,7 @@
 # Generated by Django 3.0.3 on 2020-05-15 08:01
 
 from django.db import migrations, models
-import myauth.models
+import etebase_server.myauth.models as myauth_models
 
 
 class Migration(migrations.Migration):
@@ -19,7 +19,7 @@ class Migration(migrations.Migration):
                 help_text="Required. 150 characters or fewer. Letters, digits and ./+/-/_ only.",
                 max_length=150,
                 unique=True,
-                validators=[myauth.models.UnicodeUsernameValidator()],
+                validators=[myauth_models.UnicodeUsernameValidator()],
                 verbose_name="username",
             ),
         ),

etebase_server/myauth/migrations/0003_auto_20201119_0810.py

@@ -1,7 +1,7 @@
 # Generated by Django 3.1.1 on 2020-11-19 08:10
 
 from django.db import migrations, models
-import myauth.models
+import etebase_server.myauth.models as myauth_models
 
 
 class Migration(migrations.Migration):
@@ -14,7 +14,7 @@ class Migration(migrations.Migration):
         migrations.AlterModelManagers(
             name="user",
             managers=[
-                ("objects", myauth.models.UserManager()),
+                ("objects", myauth_models.UserManager()),
             ],
         ),
         migrations.AlterField(
@@ -30,7 +30,7 @@ class Migration(migrations.Migration):
                 help_text="Required. 150 characters or fewer. Letters, digits and ./-/_ only.",
                 max_length=150,
                 unique=True,
-                validators=[myauth.models.UnicodeUsernameValidator()],
+                validators=[myauth_models.UnicodeUsernameValidator()],
                 verbose_name="username",
             ),
         ),

etebase_server/settings.py

@@ -15,7 +15,8 @@ import configparser
 from .utils import get_secret_from_file
 
 # Build paths inside the project like this: os.path.join(BASE_DIR, ...)
-BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+SOURCE_DIR = os.path.dirname(os.path.abspath(__file__))
+BASE_DIR = os.path.dirname(SOURCE_DIR)
 
 AUTH_USER_MODEL = "myauth.User"
 
@@ -43,6 +44,7 @@ DATABASES = {
     }
 }
 
+DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
 
 # Application definition
 
@@ -53,9 +55,9 @@ INSTALLED_APPS = [
     "django.contrib.sessions",
     "django.contrib.messages",
     "django.contrib.staticfiles",
-    "myauth.apps.MyauthConfig",
-    "django_etebase.apps.DjangoEtebaseConfig",
-    "django_etebase.token_auth.apps.TokenAuthConfig",
+    "etebase_server.myauth.apps.MyauthConfig",
+    "etebase_server.django.apps.DjangoEtebaseConfig",
+    "etebase_server.django.token_auth.apps.TokenAuthConfig",
 ]
 
 MIDDLEWARE = [
@@ -73,7 +75,7 @@ ROOT_URLCONF = "etebase_server.urls"
 TEMPLATES = [
     {
         "BACKEND": "django.template.backends.django.DjangoTemplates",
-        "DIRS": [os.path.join(BASE_DIR, "templates")],
+        "DIRS": [os.path.join(SOURCE_DIR, "templates")],
         "APP_DIRS": True,
         "OPTIONS": {
             "context_processors": [
@@ -138,6 +140,8 @@ config_locations = [
     "/etc/etebase-server/etebase-server.ini",
 ]
 
+ETEBASE_CREATE_USER_FUNC = "etebase_server.django.utils.create_user_blocked"
+
 # Use config file if present
 if any(os.path.isfile(x) for x in config_locations):
     config = configparser.ConfigParser()
@@ -163,11 +167,31 @@ if any(os.path.isfile(x) for x in config_locations):
     if "database" in config:
         DATABASES = {"default": {x.upper(): y for x, y in config.items("database")}}
 
-ETEBASE_CREATE_USER_FUNC = "django_etebase.utils.create_user_blocked"
+    if "database-options" in config:
+        DATABASES["default"]["OPTIONS"] = config["database-options"]
+
+    if "ldap" in config:
+        ldap = config["ldap"]
+        LDAP_SERVER = ldap.get("server", "")
+        LDAP_SEARCH_BASE = ldap.get("search_base", "")
+        LDAP_FILTER = ldap.get("filter", "")
+        LDAP_BIND_DN = ldap.get("bind_dn", "")
+        LDAP_BIND_PW = ldap.get("bind_pw", "")
+        LDAP_BIND_PW_FILE = ldap.get("bind_pw_file", "")
+        LDAP_CACHE_TTL = ldap.get("cache_ttl", "")
+
+        if not LDAP_BIND_DN:
+            raise Exception("LDAP enabled but bind_dn is not set!")
+        if not LDAP_BIND_PW and not LDAP_BIND_PW_FILE:
+            raise Exception("LDAP enabled but both bind_pw and bind_pw_file are not set!")
+
+        # Configure EteBase to use LDAP
+        ETEBASE_CREATE_USER_FUNC = "etebase_server.myauth.ldap.create_user"
+        ETEBASE_API_PERMISSIONS_READ = ["etebase_server.myauth.ldap.is_user_in_ldap"]
 
 # Efficient file streaming (for large files)
-SENDFILE_BACKEND = "django_etebase.sendfile.backends.simple"
-SENDFILE_ROOT = MEDIA_URL
+SENDFILE_BACKEND = "etebase_server.fastapi.sendfile.backends.simple"
+SENDFILE_ROOT = MEDIA_ROOT
 
 # Make an `etebase_server_settings` module available to override settings.
 try:

etebase_server/urls.py

@@ -1,25 +1,13 @@
 import os
 
 from django.conf import settings
-from django.conf.urls import url
 from django.contrib import admin
-from django.urls import path, re_path
+from django.urls import path
 from django.views.generic import TemplateView
 from django.views.static import serve
 from django.contrib.staticfiles import finders
 
 urlpatterns = [
-    url(r"^admin/", admin.site.urls),
+    path("admin/", admin.site.urls),
     path("", TemplateView.as_view(template_name="success.html")),
 ]
-
-if settings.DEBUG:
-
-    def serve_static(request, path):
-        filename = finders.find(path)
-        dirname = os.path.dirname(filename)
-        basename = os.path.basename(filename)
-
-        return serve(request, basename, dirname)
-
-    urlpatterns += [re_path(r"^static/(?P<path>.*)$", serve_static)]