simplesshd/NOTES

BUILD INSTRUCTIONS:


To build this without my private keys, you'll need to replace my
ant.properties symlink with an empty file.  You'll have to build a
"debug" apk instead of a "release" one.  Android will not let you install
a debug apk on top of a release one, so you have to remove stock
SimpleSSHD first before installing the debug build.

Then follow these steps (roughly the "doit" script):

   ndk-build -j8 &&
   mv libs/armeabi/scp libs/armeabi/libscp.so &&
   mv libs/armeabi/sftp-server libs/armeabi/libsftp-server.so &&
   mv libs/armeabi/rsync libs/armeabi/librsync.so &&
   mv libs/armeabi/buffersu libs/armeabi/libbuffersu.so &&
   mv libs/x86/scp libs/x86/libscp.so &&
   mv libs/x86/sftp-server libs/x86/libsftp-server.so &&
   mv libs/x86/rsync libs/x86/librsync.so &&
   mv libs/x86/buffersu libs/x86/libbuffersu.so &&
   ant debug

The mv steps are very important, because ant will only package the
necessary binaries if they have a .so extension (even though they are
stand alone executables).



DEVELOPMENT JOURNAL:


December 6, 2014.

The idea is to make a proper ssh implementation for Android.  Important
features:

   * it should run happily without root (on a non-root port)

   * it should be a regular android app requiring no special permissions,
     and not requiring any 'magic' executable files

   * should not rely on busybox

   * preferably support sftp

   * open source

The existing apps are either expensive, don't work, need root, or too
complicated, or a mix of all of the above.  And none of them are open
source.

I figure I'll start with dropbear, which I will run through JNI instead
of putting it in its own binary (because making such a binary executable
is a bit of a hack).

So that's the plan........


December 14, 2014.

I got dropbear to compile under the Android NDK, so now it's time to work
on the Android side of it.

I need:
  * a Service that can be started, stopped, and queried for whether it's
    running or not
  * a Thread to implement the Service's work (by calling into dropbear's
    main()), which can also be stopped.
  * a config UI with at least these choices:
    - bool: start on boot (def: false)
    - number: port number (def: 2222)
    - string: path to authorized_keys file (def: /sdcard/ssh)
    - string: name of default shell (def: /system/bin/sh -l)
    - string: default path for HOME (def: /sdcard/ssh)
    - button: start or (if it's running) stop


December 15, 2014.

Getting to the fun part.  Process management...

To start sshd, it seems like I can startService().  Then in the Service's
onStartCommand(), call startForeground() so it won't be killed (return
START_STICKY too?).

The question is if dropbear's main() should run under a separate Thread,
or a separate Process.  The trouble with a Thread is that it might be
hard to kill.  The trouble with a process is that there is no way to
report back status (such as a failure to start sshd).

Connectbot starts a new process for its shell -- it really doesn't have a
choice because the shell binary isn't linked with Connectbot, and exec()
in a thread stinks.  To stop it, it just closes stdin/stdout!!!  So
zombies can (and do) linger.

I suppose dropbear could be in its own process if it had something like
stdin/stdout to communicate failure?  Or it could just write error
messages to (i.e.) /sdcard/ssh/log.  To stop the service, it would just
use kill().

I am curious how the main waiting-for-connections loop looks, but even if
it uses select(), I'm not sure how I would honor Thread.interrupt() or
whatever.  It's not guaranteed to interrupt select(), and I'm not keen on
adding an arbitrary timeout/polling feature to it.


December 20, 2014.

So, I added a builtin scp endpoint.  It was pretty straight forward,
except dropbear defaults to vfork(), which blocks the parent until the
child runs execve()!!

Anyways, I noticed that scp doesn't quote its arguments to the remote
scp. That means you can't conveniently copy a remote file with a space in
its name (it becomes two files).  But the upside is that this is where
wildcards are handled -- by the shell!

So I need to either run it as a separate executable launched through the
shell, or make my own implementation of wildcards.

It is easy, using a $(BUILD_EXECUTABLE) script, to get ndk to build an
executable.  But it is only packaged up if it is named "gdbserver" (and
debug apk), or "libfoo.so".  The good news is that libfoo.so can be
executed in /data/data/org.galexander.sshd/lib/libfoo.so, so that is a
viable option.

Doing the expansion myself is not necessarily hard either, though.  I
need a library function called glob(), which is apparently not part of
bionic.  But I have the idea some cut and paste would resolve that with
very little extra work on my part.


December 21, 2014.

Well, bionic libc *does* provide fnmatch(), and even scandir() (a
shortcut for readdir).  In the best case, though, that still leaves me
with a bit of a path parsing conundrum (I have to tell scandir which
directory to operate on).  And also a bit of an escape character
conundrum -- \* and "*" should not act like wildcards.

Those are not insurmountable but I think I've talked myself out of it.
So then the question is, do I figure out how to ship an executable, or
do I do some hack like open a pipe to "/system/bin/sh echo filespec" and
use the shell solely for expansion?

I'm developing the idea that it's actually pretty easy to ship an
executable, I just need to find some -pre-package step where I can do
"mv scp libscp.so" and then it will ship.  ndk-build will not let me make
a target with a "." in it directly.

...

Now scp, sftp-server, and rsync work as separate executables... rsync
does fail at -z because it needs it's own custom zlib...The stock
"external" one seems to lack the "old-style compress" method.  There is
another commandline option for the new (deflate?) technique, but I think
the normal -z ought to work too.  But that is literally the last feature!
Then just release details.


December 29, 2014.

First problem report from a user.  Lollipop (Android 5.0) requires "PIE"
executables -- position independent code.  I think that is a modern
equivalent to -fpic that Android is now requiring so that it can
randomize addresses to try to obscure stack smashing attacks that rely on
fixed addresses.  It is epicly lame.

Anyways, the big fuck-you from Google is that Ice Cream Sandwich (Android
4.1) and earlier require fixed-position code.  So one binary will not
generally work on both.

Here is a good summary:
   https://code.google.com/p/android-developer-preview/issues/detail?id=888

There is something called "run_pie" which you can wrap your executables
in that lets older Android run PIE executables.  It would require a
relatively small change to the exec() call to prepend it with "run_pie".
That seems like a hack.

The suggested remedy is to build two different apks!  Yuck!

Anyways, it is only executables (not libraries -- they are position
independent already) that are affected.  And apparently static
executables don't care one way or the other.

So that is my remedy -- static executables for the moment.  I tested them
and it is only a little bit bigger -- 904kB of binaries instead of 668kB.


January 18, 2015.

Markus Ethen suggested it display the current IP address so you know
where to ssh to, in case it isn't convenient to use static dhcp or to
remember the address.  That seems to be easier said than done.  You can
use WifiManager, but that won't give your IP address unless you're on
wifi.  That is probably "good enough", but it is certainly not ideal.
There is also java.net.NetworkInterface, which seems to return a random
ipv6 address.

Ah-hah!  It is fe80::macaddr, which is a bogus "local-connection only"
ipv6 address, like 192.168, but automatically-generated without dhcp.
So if I skip that, it finds the proper ipv4 address!

Pfew!  I was thinking I'd have to directly use /proc/net/dev and
SIOCGIFCONF, just like ifconfig does, but it works fine with
java.net.NetworkInterface.


June 20, 2015.

At some points, rsync is only write()ing, and assumes that the other end
will receive it all.  The other end does a little bit of write()ing of
its own, and then is happy to read() all it wants.  So this written stuff
may sit in a buffer somewhere indefinitely.  If that happens, an
infelicity in the design of SuperSU causes everything to wedge.

Of course, this is only if you set the shell to /system/xbin/su as a way
of having root access for rsync.

Anyways, I made a new program, "buffersu", which is just a
stdin/stdout-buffering wrapper for rsync that is guaranteed to always
perform any read() that is possible at any time, no matter how many
write()s are outstanding.  That seems to do the trick.


June 21, 2016.

Chris Moore reports that rsync and sftp do not like files larger than 2GB.

rsync was easy - it just needed an additional #define in rsync/config.h
to enable its builtin support for using stat64/lseek64/off64_t/etc.

Now that I'm investigating sftp, I find this surprising fact about bionic
(though the glibc man page for stat(2) tried to tell me this) - stat64
and stat are the same thing!  But off64_t and lseek64 are significant.
That should make converting sftp pretty convenient.  Especially since
sftp already uses "u_int64_t" instead of off_t.

p.s. Chris Moore gave me this command to test sftp, which turned out to
be useful:

   curl -v --pubkey .ssh/id_rsa.pub -r 2147482624-2147484672 -k sftp://mushroom:2222/sdcard/ssh/buh -o buh-new

As for scp, it's not as clear what needs to be done.  It doesn't use
lseek.  But it does use off_t a bit, including on an index in a for loop
that is compared against st_size (which is 64-bit).  So I'll just change
all of the off_t to off64_t and hope for the best.

sftp and rsync work!  Not gonna bother testing scp on big files...


October 1, 2016.

Jared Stafford told me startForeground() improves responsiveness on
Nougat.  There had been a comment suggesting startForeground(), but I
never got around to trying it because it has worked "well enough".  With
Nougat, though, there is a definite tendency for SimpleSSHD to be
non-responsive.  I'm not sure exactly what its cause is, but the symptom
I notice most frequently is that the first ssh connection after a while
will be delayed "a long time" - on the order of 10-30 seconds, or maybe
indefinitely sometimes.  Oddly, a second connection can sometimes get
through undelayed, even before the first connection does.  It is as if
the fork() of process for the new connection is where the delay is, not
in the listen() call.

That's not overall too surprising, Nougat is a lot harsher about
background processes as part of a Google push to reduce power consumption
on idle devices.

Another concern is related to a change back in July - sometimes the
system will kill the sshd process for no good reason (maybe because they
removed it from the recent apps list).  The remedy I settled on was to
monitor for the sshd process dying from within the regular
Android-managed process, and restart it.  I guess I didn't write down
where in the documentation I found it, but Android seems expressly
antagonistic to non-system-managed processes.  If they ever get more
hard-assed about that, this whole idea goes out the window.

Anyways, I implemented startForeground(), and I am unhappy that it
requires a Notification. At API 7 (Android 2.1) those are really
primitive.  For example, the PRIORITY_MIN behavior which will sometimes
hide the notification is added in API 16 (Android 4.1).  So, I've got it
with this stupid old-style notification, and it really doesn't look good,
and it is always present.  That is not awesome.

On the other hand, it's easy to block the notifications, and a few people
have expressed an interest in a notification.

It doesn't seem worth it to me to upgrade to a newer API just for the
better notifications...  On the other hand, Google Play shows that I have
862 users:
    Android 7+  :  3.13%
    Android 6+  : 37.47%
    Android 5+  : 67.86%
    Android 4.1+: 96.62%
    Android 4.0+: 98.01%

The oldest reported version is Android 2.3.

So, there are a few people on very old versions, but actually SimpleSSHD
is used on newer devices than the average app, which is the reverse of my
typical trend.  So a few people would be negatively impacted, but not a
very large number.  I could switch to multi-APK mode so that legacy users
are just stuck with an unsupported back-version, which is probably what
they truly want anyways..

Anyways, I'm gonna use it with startForeground() and the notifications
disabled for a while, and if I find it to be an improvement then I'll
just make it so clicking on the notification goes to the app, update the
doc, and then publish it.

...

Looking up doze mode details for improving a different app, I stumbled
across this:

   https://www.bignerdranch.com/blog/diving-into-doze-mode-for-developers/

   One last case which is not mentioned in the Android documentation is
   an app using a foreground service. Any process using a foreground
   service is exempt from Doze Mode effects, which is key for
   long-running applications that you may want to run even if the user
   has put their phone down for a long period of time -- like a music app,
   for example.

   There is, however, a bug in Android Marshmallow which requires a
   foreground service to be in a separate process from all other activity
   code, documented here. It is planned to be fixed in Nougat.

Not exactly confidence-inspiring.  And I haven't come across a
description of how exactly doze mode screws up SimpleSSHD, either.

It might be worthwhile to do the work to find out where things actually
get wedged, it might even be a flaw in Dropbear that makes it so
unreliable with Nougat.


October 16, 2016.

I've been using a foreground service for a while, and it has not caused
me any troubles.  The thing I have been most anxious about is that it
might disable doze mode entirely...but there has not been any noteworthy
overnight drain.

So, in case someone does have problems with it, I am making it an option
that defaults to enabled.

Looking through email, I haven't seen any, but I have the idea several
users have asked for a notification icon in the past.  And now that that
is finally implemented, I am curious about other things people have
requested that I have not been keen on.  And Jan Ondrej's requests come
to mind:

  o) setting to start the service automatically when the application is
     launched
  o) QUIT button that stops the service and the activity at once
  o) not allow wifi to power down when the activity is open

I'm really not crazy about integrating any kind of wakelock.  And having
two buttons still seems silly to me.  But with the notification there,
the idea that someone will micromanage whether the service is running or
not does not seem so far fetched.

So I'll go ahead and add "Start on Open" setting.

...

Looking at reviews on Google Play Store, I finally found a couple reviews
in the last couple months who would have enjoyed password login.  I'm
still pretty opposed to it because it seems like the usage model would be
to enable a default password, login with it to transfer an
authorized_keys file, and then disable the default password.  That is
fine, and would even be kind of convenient (I might use it), but it seems
more often than not the default password would be left enabled
indefinitely, which is not awesome.  And I can't imagine more than 1% of
users ever typing a strong password into their phones.

But I've thought of a compromise.  What if, if there is no
authorized_keys file, it accepts passwordless logins but with some sort
of obnoxious alert dialog sort of thing to interrupt the user?  That way,
you would be able to login once to copy the authorized keys file, and the
nuissance alert would be no big deal.  Then once the authorized keys
exists, no further action is necessary.

It's a pity there is no convenient way to interact between the Android
GUI thread and the sshd thread.

Ah-hah!  I've got the least effort idea in my head!  Under situations
that I'm not sure what they should be, it should generate a random
password and display it on the phone's screen.  Probably it should
generate it iff there is no authorized_keys file at all.  Since that
detection happens for each client connection, then probably all of this
logic should go in dropbear itself, and the display should come through
dropbear.err.

Then we might even want a UI way to delete authorized_keys, perhaps even
as a replacement for the current awkward UI.


November 19, 2016.

I got a user request to update for security.  I looked at dropbear and
didn't see any relevant security issues, so I'm gonna hold off for a
while, but it should be on my radar.

I also got a user request for writing to external SD card.  He gave
this link:

   http://stackoverflow.com/questions/33162152/storage-permission-error-in-marshmallow

It gives me a really strong deja vu, I think I tried that a few months
ago when a nearly identical request came in, and it didn't work.  I wish
I had kept notes for that experiment!

I think there are two separate issues.  I think Android 6+ push you to
use requestPermission() to explicitly enable the WRITE_EXTERNAL_STORAGE
permission, but if targetSdkVersion is low enough then you are
grandfathered in, so we don't care.  The second issue is that Android 5+
require use of a special API to access a removable SD card (which is
different from /sdcard, which is typically internal storage protected by
WRITE_EXTERNAL_STORAGE on new phones?).  That second issue is what is
biting people.

I don't know any way around that.  If Google doesn't back down, then I
guess the only plausible way around it would be something like an
LD_PRELOAD that intercepts open/lseek/read/write/close to external SD,
and replaces them with connection to a local daemon process that is
running in a typical Android context and is able to use the awful API.

Seems like a lot of work and complication.  I might go through with it if
I happened to use external SD myself, but I'm of the personal opinion
that removable storage is obsolete now that even relatively cheap phones
like the base Moto G4 come with 16GB...  640kB ought to be enough for
anybody.

I told the most recent guy to try SuperSU.  I don't have any idea if that
will really work, to be honest.


October 28, 2017.

At the beginning of October, a user notified me that rsync doesn't work
on Android 8.0 (Oreo, API 26).  In the past week, Google has upgraded my
Nexus 5x to Oreo and I can confirm it.

It seems that SE Linux or something causes certain system calls to
perform the equivalent of:

    fprintf(stderr, "Bad system call\n");
    exit(-1);

It should return ENOSYS instead, but someone at Google is not cool enough
to be working with Unix.

The first one I discovered like this is sigprocmask(), which we can
probably do without.  The next one is chmod(), which is somewhat more
useful.

The troubling thing is that I can find a file where the commandline chmod
works, but the same chmod() doesn't work in rsync.

I figured maybe my NDK was just too old ("r10d"), but I think I can't
conveniently upgrade it because Google only distributes the NDK for Linux
x86-64 now.  I was using SDK 7, so I tried SDK 11, 17, 19, and got the
same result.  The NDK I have also has a directory for SDK 21, but it has
typos in the header files, or they aren't compatible with its version of
gcc.  But anyways, if SDK 19 doesn't work then maybe it's more than just
linking against a bad version of Bionic.

Well, I think I've got it - fchmod() works, but chmod() does not.
*shrug*


March 4, 2018.

Looked in the google play console for crashes, and I find a few.  10
people have run into UnsatisfiedLinkError, and they seem to all be using
x86.  According to

https://stackoverflow.com/questions/32598756/android-ndkhow-to-exclude-x86-device-in-google-player

It's not plausible to fix this problem by blacklisting x86 devices!
Google sucks!  And the justification is that there is an ARM emulation
layer, but obviously it doesn't work and why should I have to debug it??

Anyways, someday may want to try making a jni/Application.mk and setting
APP_ABI in there to support x86 too?  Do I really think that will work
for scp/rsync/etc?  Guh.

And this bug has only one report...  The startActivity() call to open the
documentation is getting a android.content.ActivityNotFoundException.  I
guess it must mean there is no browser installed or something?  I seem to
be using the normal way to visit an http programatically..  Only one
report, so I guess I just don't care.


March 24, 2018.

Win Bent emailed me to let me know his x86_64 Asus ZenPad is able to run
SimpleSSHD but is unable to run rsync.  So, the ARM-on-x86 emulation
layer is catching up and now works sometimes, but isn't compatible with
the hack for scp/sftp/rsync.  Win was kind enough to test a build that
supports just armeabi and x86, and reported it works.  So I don't need to
build support for both x86 and x86_64, apparently.  Nice to have a
concrete answer to the question!



May 16, 2018.

Roland Jaeger let me know busybox's "su" doesn't appreciate being called
with "-" prefix to argv[0] for a login shell, it can't find the "-su"
applet.  su wants "-" to come in argv[>=1] instead to indicate a login
shell.

I guess it's kind of a hack but I just don't see it causing much
trouble...  If the string "su" is found in the shell's name, it runs
"su -" instead of "-su".  *shrug*.  Roland tested it and reports it works
for him.

He also notes that the default /system/bin/sh distributed with his device
doesn't run .profile, instead it runs whatever is in $ENV.  Apparently
mksh is "the Android shell", and indeed that is what's on my Nexus 5x.
mksh is supposed to run /etc/mkshrc (well, /system/etc/mkshrc on Android,
I guess), and then $ENV.  But on my device, it loads .profile too.  So I
guess there is some diversity of configurations.  *shrug*

NB - bash supports the same idea but it calls the variable $BASH_ENV, or
$ENV if --posix is set.  *shrug*  And whatever shell comes with
SimpleBusyBox (ash?) loads .profile by default.

Anyways, it seems the obvious way forward is to make a setting for the
environment...just one name=value per line.

...

Noticed a couple crashes in the Play store console.  One is a repeat of
this Note 7 that gives ActivityNotFoundException when viewing
documentation.  I decided to catch the exception and display a dialog
asking the user to contact me, because I'm curious.

The other crash is a NullPointerException due to a bug in older Android
(a 2013 Nexus 7).  It isn't my fault and I don't care.
   https://issuetracker.google.com/issues/36972466

There is also an ANR, I think it's my very first!  Here's the message:
   Input dispatching timed out (Waiting to send non-key event because the
   touched window has not finished processing certain input events that
   were delivered to it over 500.0ms ago. Wait queue length: 25. Wait
   queue head age: 34478.4ms.)
I can't tell if it's saying it blocked for 500ms or for 34s, but either
one seems problematic.

It's 4 identical-looking failures on one day, so it might be a fluke.
It's on x86_64, though I don't think that's the problem...

There are a ton of threads so it will be a bit of a trick to try to describe:
   main - waiting in epoll_wait() under a bunch of framework
          MessageQueue/Looper stuff.
   Thread-23 - sleeping in UpdaterThread.run() line 26
   Thread-15 - SimpleSSHDService.run()'s call to native waitpid()
   and just a ton of misc threads that don't seem interesting to me

So, it seems like the main thread is specifically waiting for something
to do, I have no idea why it's not responding to user input.  I don't
think the blocked UpdaterThread or SimpleSSHDService thread, because
those are specifically in other threads because they are designed to
block.  I'm thinking it's a fluke of some sort, maybe a DoS sort of
scenario on this one device.

Anyways, the other thing I found out at the Play store is I got warned
because I'm supposed to "target a recent SDK", which means API level 26
(Android 8.0 Oreo).  I think this is dumb but apparently it only means
I have to update android:targetSdkVersion to 26.  I can leave
minSdkVersion at 7, and target=android-7.  I hope.

I imagine a bit of UI will move around but it should just be a matter of
shaking it out and then testing it on an older device.

Oh, I just remembered the important new Androidism - permissions.  In
particular, I think it is now more complicated than just
WRITE_EXTERNAL_STORAGE.  Ugh?

Using my Nexus 5x (Android 8.1), updating to a new one with
targetSdkVersion 26, everything seems to just work.  But the
blue-on-white for the IP display becomes blue-on-black and it is almost
unreadable.  And also, it is kind of ugly.  I think if I can switch it to
a dark theme or something that will work itself out.

But then I uninstalled the app and installed it from scratch with
targetSdkVersion 26, and that does not work at all.  On startup it shows
the next "null" on every line where it's supposed to show the log.  And
once the service is started, the log is still null, and anything
attempting to connect says:
   Connection closed by 192.168.1.23

So my guess is that it's now more complicated than just
WRITE_EXTERNAL_STORAGE.  Ugh!

...

A little info from

https://medium.com/google-developers/picking-your-compilesdkversion-minsdkversion-targetsdkversion-a098a0341ebd

which told me I can put the compile target much ahead of the min target
and supposedly I'll get warned if I use new features.  Anyways, maybe I
can gate the new permissions model behind a simple if?  I'm not sure if
it will be enough to explicitly ask for the permission, or if the
permission is accessed differently.

It also suggests I look at the "Platform Highlights" in the API Level
table at:

https://developer.android.com/guide/topics/manifest/uses-sdk-element

That's quite a few to check out to go from SDK 11 (Honeycomb) to SDK 26
(Oreo).  But I'll probably learn something...

Have to use checkSelfPermission() and requestPermission() to get
"dangerous" permissions (WRITE_EXTERNAL_STORAGE).  Luckily, INTERNET and
RECEIVE_BOOT_COMPLETED are "normal" permissions and just need to be in
the AndroidManifest.xml, I guess.  And I don't need to worry about
READ_EXTERNAL_STORAGE, so long as I explicitly request
WRITE_EXTERNAL_STORAGE.


May 19, 2018.

Michael Mess wrote to let me know that he can't use authorized_keys file
because the /sdcard/ssh/authorized_keys file is group-readwritable.  I
know regular OpenSSH on a PC does complain if the authorized_keys file is
world-writable, so this made sense to me.  He says it is a security flaw,
but really, it is not, because any app that can run code and write to
/sdcard/ssh already has all of the permissions that SimpleSSHD has
(except network, but whatever) -- it isn't escalation, just a horizontal
spreading.

He suggested moving the SSH config into app-private, which is something
I've been toying with anyways.

So after some internal debate, I decided to do two changes:

  * add "Copy app-private path" to the options menu, which brings up a
    dialog showing the app-private path, with an option "Copy" to put it
    in the clipboard.
  * make the default for new installs for "home" and "path" be the
    app-private dir

I look forward to seeing if anyone complains about it.


May 20, 2018.

I tried updating the build target to a newer version than minSdkVersion,
and I got a warning, and I looked at the official google documentation,
and you can't build with anything newer than the minSdkVersion.  I mean,
I knew that, so, what I'm saying is, I feel stupid for reading that
medium.com link.


June 8, 2018.

Request from Joan Marce i Igual for --iconv.  That requires iconv.h,
which isn't part of Bionic.  GNU libiconv will do it, but it doesn't look
particularly easy to build, and it isn't small.  Someone on github has
uploaded their tree for building iconv on Android, which might save me a
bunch of effort:

   https://github.com/ironsteel/iconv-android.git

Unfortunately it doesn't have much history to it, so I don't know if
it'll be worth anything or not but it should give me a sense of the build
task without forcing me to decipher autoconf.


May 25, 2019.

To update to the new SDK (26 required now, 28 required starting in
August), I have to switch to gradle and all that garbage.  Every time I
update anything with the new SDKs, everything breaks, even after I've
already updated everything to gradle.  So basically it is the worst
possible development environment.

Anyways, the NDK now doesn't support anything older than android-16 (JB
4.1), which is stupid but whatever.  Looking on the play store, it looks
like I have 5 users on these older devices.  I hope it simply refuses to
update for them, and they can continue to use the older version!

... I got it to build and install and run, and it "works".  The only
issue that is currently biting me is (on using am to install an apk):

   InstallStart: Requesting uid 10180 needs to declare permission
   android.permission.REQUEST_INSTALL_PACKAGES

Anyways, there's a bunch of stuff remaining to do, and then I will
release it.


May 26, 2019.

I got an AVD (emulated android) running SDK 26 (Oreo 8.0) x86.  It uses
kvm virtualization so it's much faster than the old emulated ARM I used
to be shackled with.

Anyways, right off, it says (in a toast):
   Developer warning for package "org.galexander.sshd"
   Failed to post notification on channel "null"
   See log for more details
adb logcat says:
   05-26 17:13:49.877  1741  1764 E NotificationService: No Channel found
   for pkg=org.galexander.sshd, channelId=null, id=1, tag=null,
   opPkg=org.galexander.sshd, callingUid=10085, userId=0, incomingUserId=0,
   notificationUid=10085, notification=Notification(channel=null pri=0
   contentView=org.galexander.sshd/0x7f030001 vibrate=null sound=null
   tick defaults=0x0 flags=0x40 color=0x00000000 vis=PRIVATE)

This gives a kind of deja vu -- I think from BluntMP3?  I'm just adding
it to the todo list.

To access SimpleSSHD, run
   adb forward tcp:2222 tcp:2222
   ssh -p 2222 localhost

And it demonstrates that the sdcard access is forbidden.  In fact, I can
toggle it inside the System Settings -> Applications -> SimpleSSHD ->
permissions -> Storage, and it takes effect more or less immediately.

Android development is such a constant stream of astonishing garbage.
When it requests the permission, which happens just as I would want it
to, it then pops up a "System UI has stopped" dialog.  But look, it isn't
my fault:

   https://stackoverflow.com/questions/49261555/system-ui-has-stopped-emulator-when-requesting-permissions

For real.  There's a bug in the emulator you fix by "increasing the DPI."
I had tried some older 854x480 sort of emulated device because I didn't
want all those pixels, but I'll just use Pixel 2 (the default, and one
Google surely tests).  It has a different skin where the soft buttons are
part of the skin, which is I bet what the bug is around.

So, the permissions request works now.  It just asks once, and no matter
what the answer is, it never asks again.  And if it is grandfathered in
because the previous version of the app used an older SDK, then it will
never ask and it'll just have the permission.  And, at least on an
emulated Pixel 2, it can access /sdcard, which is "internal storage."
It does emulate an "SD card", but it can't access that.

...

Getting a channel onto the notification was a pain in the ass.  All this
gradle stuff is getting increasingly undocumented.  Every time they
change something so they won't ever have to change anything again, they
then immediately change everything.  This leads to a mass proliferation
of immediately-obsoleted documents.  It even damages resources like
stackexchange because people take their extraordinarily unique context
for granted when asking and answering questions.  So something that
worked against gradle 3.3.4 will now be broken against gradle 3.3.5 but
no one will think to mention what version of gradle, or what version of
the build scripts gradle runs, or what-have-you.

Anyways, the upshot is, the way I got it to work is I opened the project
in Android Studio, and eventually I used Refactor->Migrate to AndroidX,
which turned out to have all manner of complications but here we go.

Speaking of Android Studio, the AVD Manager (Android Virtual Device, I
guess) is not accessible under Tools, even though it should be.  Instead,
you have to click on the icon which is a portrait phone with a green
android head on the lower-right corner of it.  Upper-right corner.
And you have to open a project before it will show you that.  Jesus.

...

Testing with pie, I am able to reproduce the problem with start-on-boot.
Here's some excerpts from logcat:

   05-26 21:24:26.251  1837  3408 W ActivityManager: Background start not allowed: service Intent { cmp=org.galexander.sshd/.SimpleSSHDService } to org.galexander.sshd/.SimpleSSHDService from pid=3756 uid=10085 pkg=org.galexander.sshd startFg?=false
   --------- beginning of crash
   05-26 21:24:26.251  3756  3756 E AndroidRuntime: FATAL EXCEPTION: main
   05-26 21:24:26.251  3756  3756 E AndroidRuntime: Process: org.galexander.sshd, PID: 3756
   05-26 21:24:26.251  3756  3756 E AndroidRuntime: java.lang.RuntimeException: Unable to start receiver org.galexander.sshd.BootReceiver: java.lang.IllegalStateException: Not allowed to start service Intent { cmp=org.galexander.sshd/.SimpleSSHDService }: app is in background uid UidRecord{5aff37a u0a85 RCVR idle change:uncached procs:1 seq(0,0,0)}
    ...
   05-26 21:24:26.251  3756  3756 E AndroidRuntime: Caused by: java.lang.IllegalStateException: Not allowed to start service Intent { cmp=org.galexander.sshd/.SimpleSSHDService }: app is in background uid UidRecord{5aff37a u0a85 RCVR idle change:uncached procs:1 seq(0,0,0)}
    ...
   05-26 21:24:26.251  3756  3756 E AndroidRuntime:        at android.content.ContextWrapper.startService(ContextWrapper.java:664)
   05-26 21:24:26.251  3756  3756 E AndroidRuntime:        at org.galexander.sshd.BootReceiver.onReceive(BootReceiver.java:11)

So it looks like a pretty intentional rule that you can't start a service
from a boot receiver??

And I have the same problem on my Z2 Force (Oreo 8.0) now that I've
updated the SDK.

This guy says it will allow you to startForegroundService(), so that's
what I'll try...

   https://stackoverflow.com/questions/49572570/android-start-a-service-on-boot-not-working

Which seems to work!  So there is a need for some sort of dance in the
Settings screen to force it to enable foreground services if
start-on-boot is selected, and then we'll be golden.


June 16, 2019.

I finished the dropbear 2019.78 merge and got it all to work again on the
x86 version 28 (Pie) emulator.  It also works on x86 version 26 (Oreo)
emulator, and on my ARM Oreo phone (Moto Z2 Force).  I also tried with an
emulated ARM version 16 (JB 4.1), and it works, but it generates a lot of
this spamming the log:

   Failed lookup: Non-recoverable failure in name resolution.

This gives me a kind of deja vu, something to do with library versions. I
have the idea it's because I built with a newer libc than what is found
by dynamic linking in older Android.  But I cannot link the main dropbear
thing with -static because it needs to be JNI (which is loaded with
dlopen()).  Apparently.

Anyways, I'm not sure there's anything I can do about that other than
reduce the log noise.  I'm not really interested in digging up a magic
older version of the NDK or anything like that.

But I do think the only reason it has -static on the stand-alone
executables is basically a hack to make them work with the old NDK, so I
can get rid of that...

Testing with emulated x86 Android 21 (KK 5.0)...  It seems to work fine.
It resolve my IP to the string "127.0.0.1", anyways.  rsync seems to
work, and start-on-boot in the background works.

Anyways, I made a naive thing to decode an IP address to a numeric
string, in case getnameinfo() fails.  It is working for me in the ARM
Android 16 now.  It generates a reasonable string even though it
presumably fails to use the library lookup.  It starts on boot as a
background service.  And rsync and scp work.  Only need to test on my
Moto X now.

...

Testing start-on-boot on my Moto Z2 Force, it works, but there is an
infelicity.  If I go into Settings and try to turn off foreground
service, it won't let me because start-on-boot is selected and I am using
Oreo.  It is supposed to show a toast to let me know this, but the toast
doesn't appear.  That's because I had previously disabled notifications
so that the foreground service doesn't have a notification.

I can make the toast appear by following these steps: re-enable
notifications (manually in Settings -> Apps & Notifications), then
re-start the service, then disable the notification (by long-tapping on
the notification itself).  Then toasts work.  I think probably it has
saved the fact that all notifications are disabled, because it only knew
one class of notification when it was using compatibility with the old
SDK...but now it knows two classes (a notification and a toast), and it
records that only one class is disabled.  I really have no idea but I
think it sucks.

...

Testing with Moto X (ARM, Android 5.1 API 22), and it works.  But it has
this infelicity...when scp or rsync is run:

   WARNING: linker: /data/app/org.galexander.sshd-2/lib/arm/libscp.so: unused DT entry: type 0x6ffffffe arg 0xb64
   WARNING: linker: /data/app/org.galexander.sshd-2/lib/arm/libscp.so: unused DT entry: type 0x6fffffff arg 0x2

These are DT types that are present in the .so which are not supported by
the kernel.  It may be because the kernel is CyanogenMod?  The types are
DT_VERNEED and DT_VERNEEDNUM.  They refer to a section .gnu.version_r,
which I guess holds two 32-byte entries which are unintelligible to me.

Supposedly I can use an "ELF cleaner" to strip out these DT_VERNEED
things.  But there's some ambiguity as to whether they're necessary or
useful.  I think I'm gonna leave that be.

More productively, I found that it won't startForegroundService() because
that was added at Oreo!

I can't seem to get start-on-boot to work, but from some of the crap in
logcat, I have the idea CyanogenMod might not be good at broadcasting the
boot event.

Now for the interesting test...I uninstalled SimpleSSHD on the Moto X
(API 22), and am reinstalling it new.  So it won't have grandfathered
permissions.  It works fine!  No surprises.  It still doesn't start on
boot *shrug*.

Now to try the same uninstall test with Moto Z2 Force (API 26).  It works
fine, and on the first start-up it does ask for permission to access
files (/sdcard).  To my surprise, it saved my Settings and my
/data/user/0/org.galexander.sshd/files.  That's actually kind of bad,
because the host key would be saved, and there aren't really many
settings to re-enter in the upside case.  Apparently this is new behavior
since I added targetSdkVersion>=23, so users haven't had their data
backed up yet and I can change this setting with allowBackup="false" in
the manifest...

allowBackup="false" took immediate effect and had no surprises...


August 4, 2019.

I finally got a dump from a user (Hammad), and it's quite distressing.
The stack trace is roughly:
   backtrace()
   sigsegv_handler()
   /system/bin/app_process64+0x2a90
   __kernel_rt_sigreturn()
   A5xContext::HwAddNop(unsigned int *, unsigned int)
   EsxCmdMgr::IssuePendingIB1s(EsxFlushReason, int, int)
   EsxCmdMgr::Flush(EsxFlushReason)
   EsxContext::Destroy()
   EglContext::DestroyEsxContext()
   EglDisplay::MarkContextListForDestroy()
   EglDisplay::Terminate(int)
   EglDisplayList::Destroy()
   EglDisplay::DestroyStaticListsMutexesAndTlsKeys()
   EsxEntryDestruct()
   /system/vendor/lib64/egl/libGLESv2_adreno.so+0x12780
   [... cut off at 16 ...]

So many questions!  I think app_process64 must be the actual C main() of
a process, responsible for branching into all the android system
libraries?  I imagine it's involved because it's somehow intercepted the
SIGSEGV and re-dispatched it to my handler?  I don't see any way we could
have branched into libGLESv2_adreno from userland, so the SIGSEGV must
come from the UI thread, I guess?  Maybe this SIGSEGV is actually the
sort of thing we'd get if we tried to call UI code from the non-UI
thread??

It looks like GLES is busy cleaning itself up, and it crashes.  Why's it
crash?  Why's it trying to clean itself up?

Hammad says there is no problem using sshd...I thought he meant that the
re-start logic is working for him but his dropbear.err has multiple dumps
in it!  The SIGSEGVs are apparently not killing the daemon.

There are no timestamps on the dumps, but it looks like they're
associated with activity anyways.  Each dump happens between "Disconnect
received" and "sigchld".  Some of them have "server select out"
interleaved into the dump, which I think is the result of Hammad running:
   while true; do ssh phone 'exit'; done
That is, it appears he starts a new connection the very instant the old
connection ends.  So the new connection comes into the server process
while the child process is in the act of dying.

The thing is, I don't see how it could possibly be getting signals from
the Java side of things, because it fork()s before setting up the signal
handling.  It's not just running in a different thread, it should be a
totally separate process.  I can test this but I don't think I'm wrong
about that.

So I guess just about the only thing that's really possible is that
there's an atexit() which survives the fork() because it isn't followed
up with an execve().  It's not caused by ARM, or even necessarily by
Android 9...the reason it doesn't show up in the emulator is that the
libGLES that registers the atexit() is vendor-supplied for specific
hardware ("Adreno").

So I need to figure out how to bypass the atexit() somehow, perhaps by
calling _exit() directly?


XXX - merge back into main branch, because I'll want to keep the dump facility
XXX - make the dump go deeper in the stack
XXX - put a crash in an atexit() to be sure it presents about this way
XXX - test re-start mechanism, which doesn't seem to work on the first try if it crashes
XXX - test bypassing that crash (register an atexit() just after the fork that calls _Exit(0))
XXX - remove the crash, remove the debug fprintfs (select in/out, sigchld)

--- new release

XXX - see if settings looks better with SDK26, if not, hack it so that the ones with strings have their states shown

XXX - ed25519?

XXX - try /data/data/com.termux/files/usr/bin/zsh as login shell

XXX - libiconv? HAVE_ICONV_H etc