This is a monster commit because to be honest, it's a monster change. It
was impossible to do it in smaller steps because things just wouldn't
compile.
We couldn't do the migration step by step because they moved to Kotlin
which was causing a lot of troubles.
Now we are all on Kotlin, so things should hopefully work just fine.
It just needs a few tiny wrappers around some public statics. I'm not
doing that because it'll sort itself out the moment we update
vcard4android and ical4android.
Networking is not allowed on the main thread, and on some devices with strict
mode on, even the creation of the http handler is enough to trigger an
exception (i.e even if not used from the thread).
This moves even the creation to a thread which fixes the issue.
HTTP requests and responses are logged when logging to file. Until now,
only the existence of requests was logged. With this change, also the
content and headers of the requests and responses is printed to the log.
Before this change we were printing added/changed contacts and groups
to the adb log. This is not a big deal on its own, but now since we
have ACRA, we share these logs on crash (if user approves) so it's
better to remove personal information to make sure it's not being
accidentally shared.
Android annoyingly kill sync managers that don't have a significant
amount of network traffic within a given minute. This means that if we
have a lot of entries to process, we may get killed by the system if we
have a lot of entries to prepare for pushing. We were sending in chunks,
for network performance, but now we make the whole process work in
chunks.
This should fix an issue reported by a user who imported a significant
amount of contacts in one go.
This is similar to the issue fixed for fetch in:
f7104bbcef
We were doing it to make sure we don't get overridden by
server changes. But we already changed this behaviour in
the past, so this call was just doing nothing and slowing
down the sync.
This was causing issues when importing from a Google account in some cases
because we were getting weird UIDs.
This was also problematic when importing from other sources that
reported weird UIDs.
This will make it easier to identify and fix crashes.
Until now we relied on user to automatically figure out if the app has
crashed and gather debug info manually. This didn't work well,
especially in places like "import" where they just assumed the import
finished successfully if there was a crash.
This change makes it so whenever there's a crash, the email app is
opened with a template email and the stack trace attached.
This should make it easier for us to detect and fix issues.
Important to note: nothing is sent automatically.
Some device manufacturers (I'm looking at you Xiaomi!) made some changes
to Android that break content providers and other background apps. This
affects a few apps, including DAVdroid from which EteSync is derived.
This change attempts to automatically detect such devices, alert users
and point them to the relevant FAQ entry.
I've already had to deal with a few bug reports stemming from this
issue, so it's good to have this handled automatically.
This addresses #22
Groups are saved as separate vCards. We removed support for groups to
speed up development and deferred adding them back until there was
demand.
There is demand now, and also, not having this support resulted in the
sync not working, not just groups not supported.
Many thanks to "359" (this user's preferred alias) for investigating and
reporting this issue.
Due to a logical issue in the code, new journal entries were added to the
local cache after they've been created locally, and not after they've
been added to the server. Under normal circumstances this doesn't pose a
problem, however when pushing to the server fails, the local cache
would have the new entries as if they were saved on the server, causing
the app to think there has been a corruption on the server (as entries
should never be removed from the server) and halt the sync.
This change makes it so the entries are saved to the local cache only
after they've been saved on the server.
Note: this was not spotted until now because it relies on an unfortunate
specific sequence of events. It only happens when creating journal
entries, and when trying to sync them successfully connecting to the
server to fetch the journal list and the content of the journal itself,
and only failing when coming to push the journals.
Many thanks to "359" (this user's preferred alias) for reporting the
issue that resulted in this fix.
I assumed the lifecycle of the fragment and the task were tied because they
are tied to the instance, but it looks like I was wrong. We need to
explicitly cancel tasks.
This patch changes the fetching so if the last fetch returned less entries
than the limit, we don't try and fetch again because we already know there
are no others left.
Before this commit we used to fetch the whole journal entry list in one
go, which caused issues in two cases:
1. On slow internet connections the download may fail.
2. With big journals: Android interrupts sync managers if they don't
perform any significant network traffic for over a minute[1],
and because we would first download and only then process, we would
sometimes hit this threshold.
Current chunk size is set to 50.
1: https://developer.android.com/reference/android/content/AbstractThreadedSyncAdapter.html
I set it to 2048 following the NIST recommendations[1] which said it was
OK, but actually, as pointed out by Dominik Schürmann, it's probably a
better idea to set to 3072.
Users who already have a 2048 key pair won't be affected, while users
who don't will have a 3072 key created for them. Users with different
key lengths can interact with each other without any issues.
1: https://www.keylength.com/en/4/
This change makes clicking on journal items in the list to show in a
separate activity. At the moment it just makes for a slightly nicer
presentation. In the future we would change it to show the data in a
nice formatted way instead of a raw dump of the vObject.
There was an issue that for the first load it would only check the url
after a redirect (if there is one), which meant that for example,
the dashboard, would open in app because you'd be redirected to the
login page.
Account migration works in most cases, though while testing I managed to
get it to fail in some rare occasions. This commit adds a check to
verify the number of contacts we thought we migrated is equal to the
number of contacts we have after migration.
If the check fails, it presents the user with a notification that opens
the relevant FAQ entry on the EteSync website.
This doesn't work well, but I'm keeping it since it's still better than
what was there before.
We have a problem that on initial sync with long enough logs, Android
kills the sync manager before completion. The reason for that is that
due to the fact that EteSync first downloads the whole journal and only then
processes it, the sync manager spends a minute without making any
network traffic, which in turn makes Android kill the sync[1].
This should probably be fixed by paginating the initial download, that
is, downloading and processing the journal in chunks, which is possibly
a good idea regardless.
1: https://developer.android.com/reference/android/content/AbstractThreadedSyncAdapter.html
It seems like there's an issue with Android that sometimes the userData
passed to addAccountExplicitly is not correctly set in the Android cache
making it return null on subsequent fetches. It doesn't always happen
because some cases clear the cache, however I can consistently trigger
it by creating and deleting an account a few times in a row.
This is an ugly workaround. For some reason a sync is called when an
account is removed. Since the main account is removed, we get an invalid
account exception when trying to fetch it.
Need to find out why a sync is even triggered and just remove it there.
Android allows only having one address book per account, so until now
users of EteSync were only able to have one address book. This was
always an annoying limitation, but even more so now that journal sharing
is implemented.
Luckily, DAVdroid recently implemented multiple account support by
creating sub-accounts for address books.
This patch is an import of the DAVdroid changes, with adjustments to
work with EteSync, and a few changes that did not make sense for
EteSync. The original commits' split didn't provide any value over this
squash, and the amount of adjustments and addition needed to be done to
apply them, made me decide to squash this change together.
This commit is mostly based on:
dfec72ce6b8ff5e0780e9ac4418c81d080f4b60b
9817594da14ad8dffae18de386e14aeaf41312b9
Thanks to @dschuermann for the suggestion. This makes it easier for
people of non-latin speaking cultures to compare the fingerprints.
Code is based off of Signal's fingerprint generation.
This change only works for calendars at the moment, because we don't have shared
address books anyway.
This is currently only implemented in the client, and only as a read-only attribute,
you can't make a journal read-only yet. This requires server support that is not yet
there, but it's better to be ready for this sooner rather than later.
Some of the information is now saved there, and more will be transferred
soon. CollectionInfo includes the encrypted part, and journalentity the
non-encrypted part of the journal info, so both are needed.
The reason for that is that before version 2, all the journals of a
particular user shared the same encryption key, which means, sharing a
journal of version one, would essentially give away the encryption key
of all of its journals, even the private ones.
This is thus blocked for security reasons.
This currently just adds the journal back, but doesn't re-apply the
journal, so the calendar for example would be empty, but the journal
itself would be listed and visible.
At the moment we only support one address book per user, and sharing
address books will interfere with this model. Hopefully, we'll add
multiple address book support in the next release, and then we'll
re-enable this.
We were not detecting the version correctly, but always just assumed
latest version, which is obviously wrong.
In addition, before this commit we used to automatically verify on
fetch, which wasn't flexible enough for some use cases. This fixes that
too.
We need the keypair to access shared journals, so we need to make sure
to fetch it at the moment we create the local account, which is what
this commit does.
This is used to create a keypair and put it on the server if one doesn't
exist, and fetch it and save it locally if one does.
It's currently called from the account activity.
If a journal has a key set to it (usually used for shared journals), use
it instead of the symmetric key. The key of the journal is asymmetrically
encrypted using our keypair.
Requery doesn't automatically update column constraints, and there was
an issue with it applying indexes before adding the new columns which
was also causing troubles. This commit, while ugly, just manually
updates the database using raw SQL to what we expect it to be.
Having it in raw sql was slowing down development, and was error-prone.
It's much cleaner now, easier to handle, and enables us to develop
faster.
In this change I also fixed the fetching of journals to be by service
and id, not just id, because just id is not guaranteed to be unique.
Before this change, uid was unique on its own, this was wrong, because
due to shared journals, we can have the same journal in two accounts,
and we can thus have both journal and entry UIDs more than once.
This fixes the constraint to be unique for journal, uid, and service,
uid combinations.
This is currently disabled for journals because of a bug in requery.
The server was changed so the owner of the journal, and the encrypted
key (if a shared journal) would be exposed. This change fetches it, and
saves it.
Although this release is claimed to fix the afterLoad issue, this is not
the case. We are just updating it so the upgrade path later one would be
easier.
Unfortunately this requery version introduced a regression. When adding
a new account, it takes syncign a few times until it works. It looks
like requery is not loading the recently saved instances.
This reverts commit f0f70ff1c61996d0e45d8f72d24654c739c325f7.
Prior to this version of requery there was an issue that prevented
afterLoad to be called in some cases. This issue forced us to add an
explicit call to afterLoad. It's now fixed, so the workaround is no
longer required.
Reference issue: https://github.com/requery/requery/issues/487
We were storing the ctag separately although the data was already
present in the journal. The last entry's ID is always the CTAG.
This could cause issues if sync is aborted exactly at the right time.
I managed to trigger this issue on rare cases.
It's only used for migrations, and has been considered deprecated for a
while. Mark it as deprecated to make it extra obvious that this should
not be used.
* move file name/UID generation from SyncManager to LocalContact, LocalEvent, LocalTask
* rename updateFileNameAndUID() to prepareForUpload()
* use random UUID for contacts, UidGenerator with Android device ID for events/tasks
* LocalEvent.prepareForUpload(): use existing UID_2445 if available
This also adds an icon (that will soon be replaced with the icon of the
relevant account), and shares the design between the calendar and the
contacts.
This set of commits add import from local accounts.
It's a bit rough around the edges, but it's good enough to go in, so work can continue
collaboratively.
The crypto class now behaves differently depending on the version of the
journal.
The current difference is in the key derivation, and that the new
version of the crypto also hmacs the version automatically whenever it
hmacs anything.
The versioning was added for better future-proofing of the code.
The derivation change was done because before we were creating the same
password for all of the journals, now we do it per-journal. This means
that we can, if needed in the future use this password as the journal
password when sharing journals without compromising the security of the
rest of the journals.
The app update broadcast receiver is only called on the first update,
not install, which was causing EteSync to think it was updating from
version 1 on the first update, doesn't matter which version one was
updating from.
This fixes it by saving the version on the first run.
Before this change we were adding them to cache before they were
processed, potentially persisting malformed entries, or entries we
haven't yet processed causing issues if sync was aborted before an entry
was fully processed.
Commit 5d1c90dcba fixed a bug where
entries added from the server were marked as "local only" (null etag)
which was causing issues. That commit fixes it for newly added resources,
but existing resources remained broken.
This commit goes through the database and fixes all of the existing
broken resources. It skips dirty entries because figuring out if they
were just created or updated is complex, and the chances of doing an
update at exactly the same time there are dirty entries is quite low,
so the complexity involved is just not worth it.
