Monthly Archives: February 2013

Split Contacts List (Optional)

A couple of people were asking me for something like SlidingMenu, but it think SlidingMenu can get a bit difficult for some people to use. What I’ve done is created something similar to Google Play’s partial fragment screen. I”m not entirely sure how other people are doing it, but I got it to work pretty reliably without a major hassle with ViewGroups and the like. Let me start off by reiterating that almost all display options on the application are optional.

By default, I believe I’ll set the margin between 48dp and 96dp (so… 72dp?). For this example, I’m showing the the Light theme, which is pretty much the same thing as the dark theme, with the opposite Holo colors (instead of holo_blue_light, holo_blue_dark). I’m not sure if it’s as pretty as dark. Maybe it’s the colors or just the fact I prefer black. Maybe it’s the fact, by default Light background doesn’t use a gradient for a background.

I’m setting the dp to 144dp to really show off the effect. I’ve written a custom SeekBarPreference and it looks like this:

Screenshot_2013-02-28-16-42-48

The minimum is 0dp (off), max is 192dp and increments are 8dp.

The XML for this preference is written like this:

<org.shortfuse.hermes.SeekBarPreference
  android:defaultValue="0"
  android:key="pfContactsListRightMargin"
  android:max="192"
  android:text="dp"
  android:title="@string/pfContactsListRightMargin_title"
  seekBar:increment="8"
  seekBar:min="0" />

So it’ll pretty easy to make a seekable preference from now on.

Here’s how it looks in portrait view. Just remember I’m exaggerating here for effect.

split contacts screen

Here’s with the option off and in dark:
contactsfullwidthdark

The color on the right hand side of the contact is the “Unread Indicator.” I figure having the number of unread messages seems kind of silly, since regardless, when you flick over, they will be marked red. The color is based on the IM service. Seems silly now for just SMS and GVoice, but when you start adding more services it won’t be as needlessly colorful. I’m not sure if I want to move this to the immediate right of the contacts icon instead all the way on the lefthand, since I don’t like the colors clashing on the right hand side.

I just realized I haven’t considered putting the last message time in the contacts list.

Oh well, I’ll think of something.

Mock ups and suggestions welcome.

Edit: Portrait is okay, but totally awesome on landscape. I also moved the color indicator to the other side with gray as no new messages. Check it out:

landscape split contacts

You can manipulate both lists simultaneously. Slide to the right and you’ll see the text entry box like in the other pictures.

This means forwarding messages is going to be a lot easier, since you can select/copy messages on the right and hold and “paste & send” to the contacts on the left.

You just slide to the right to get to your edit box.

landscape split motion

And finally you’ll reach your normal messaging window:

landscape split end

SQL Update

Just going to post the new changes I did because Sender/Recipient is redundant with IsIncoming. I changed it to use ExternalAddress and indexed it.

private static final String DATABASE_CREATE =
  "CREATE TABLE Messages "
  + "("
    + "MessageID INTEGER PRIMARY KEY AUTOINCREMENT, "
    + "ProviderID INTEGER NOT NULL, "
    + "MessageStatusID UNSIGNED INTEGER NOT NULL, "
    + "IsIncoming BIT NOT NULL, "
    + "InternalAddress NVARCHAR(128) NULL, "
    + "ExternalAddress VARCHAR(128) NOT NULL, "
    + "CreationDateTime DATETIME NOT NULL, "
    + "LastSendAttemptDateTime DATETIME NULL, "
    + "CompletionDateTime DATETIME NULL, "
    + "MessageText TEXT NOT NULL, "
    + "ExtraData BLOB NULL, "
    + "ExtraDataTypeID UNSIGNED INTEGER NULL, "
    + "ImportMessageID NVARCHAR(40) NULL, "
    + "ImportConversationID NVARCHAR(40) NULL, " 
    + "IsRead BIT NOT NULL" 
  + ")";

private static final String DATABASE_INDEXES_CREATE =
  "CREATE INDEX IX_ProviderID_ExternalAddress_CreationDateTime ON MESSAGES "
  + "("
    + "ProviderID ASC, "
    + "ExternalAddress ASC,"      
    + "CreationDateTime ASC" 
  + ")";
  
String rawQuery = ""
  + "SELECT "
  + "  M2.MessageID, "
  + "  M2.ProviderID, "
  + "  M2.MessageStatusID, "
  + "  M2.IsIncoming, "
  + "  M2.InternalAddress, "
  + "  M2.ExternalAddress, "
  + "  M2.CreationDateTime, "
  + "  M2.LastSendAttemptDateTime, "
  + "  M2.CompletionDateTime, "
  + "  M2.MessageText, "
  + "  M2.ExtraData, "
  + "  M2.ExtraDataTypeID, "
  + "  M2.ImportMessageID, "
  + "  M2.ImportConversationID, "
  + "  M2.IsRead, "
  + "  LastMessageData.ProviderID, "
  + "  LastMessageData.ExternalAddress "
  + "FROM "
  + "  ( "
  + "    SELECT "
  + "      M1.ProviderID, "
  + "      M1.ExternalAddress, "
  + "      MAX(CreationDateTime) [LastMessageTime] "
  + "    FROM "
  + "      Messages M1 "
  + "    GROUP BY "
  + "      M1.ProviderID, "
  + "      M1.ExternalAddress "
  + "  ) AS LastMessageData "
  + "  LEFT JOIN Messages M2 ON  "
  + "    M2.CreationDateTime = LastMessageData.LastMessageTime "
  + "    AND M2.ExternalAddress = LastMessageData.ExternalAddress "
  + "    AND M2.ProviderID = LastMessageData.ProviderID "
  + "ORDER BY                   "
  + "  LastMessageData.ProviderID, "
  + "  LastMessageData.ExternalAddress,                "
  + "  M2.MessageID";

I don’t think there’s any theoretical way to make this faster. You’re all welcome to try, and I’ll patch it in.

Raw SQL queries and why you should care

Almost every android application uses SQLite in the background. It’s pretty powerful if you know what you’re doing. I’m going to show you what well done SQL script does for performance.

This is the old getContactList function:

public List<ContactItem> getContactsList() {
  Cursor senderCursor = db.query(DATABASE_TABLE, new String[] {
      "ProviderID", "Sender", "MAX(MessageID)" },
      "Sender IS NOT NULL AND IsIncoming = 1", null,
      "ProviderID, Sender", null, null);
  List<ContactItem> contactList = new ArrayList<ContactItem>();
  if (senderCursor != null) {
    while (senderCursor.moveToNext()) {   
      ContactItem c = new ContactItem();
      int providerId = senderCursor.getInt(0);
      String sender = senderCursor.getString(1);
      long messageId = senderCursor.getLong(2);

      c.setDisplayName(sender);
      c.setDisplayContactAddress(sender);
      ContactAddress ca = new ContactAddress();
      ca.setDisplayAddress(sender);
      ca.setParsedAddress(sender);
      ca.setIMProviderType(IMProviderTypes.values()[providerId]);
      c.setContactAddresses(new ContactAddress[] { ca });
      c.setLastMessageItem(this.getMessageItem(messageId));

      contactList.add(c);
    }
  }
  senderCursor.close();

  Cursor recipientCursor = db.query(DATABASE_TABLE, new String[] {
      "ProviderID", "Recipient", "MAX(MessageID)" },
      "Recipient IS NOT NULL AND IsIncoming = 0", null,
      "ProviderID, Recipient", null, null);
  if (recipientCursor != null) {
    while (recipientCursor.moveToNext()) {
      ContactItem c = new ContactItem();
      int providerId = recipientCursor.getInt(0);
      String recipient = recipientCursor.getString(1);
      long messageId = recipientCursor.getLong(2);

      c.setDisplayName(recipient);
      c.setDisplayContactAddress(recipient);
      ContactAddress ca = new ContactAddress();
      ca.setDisplayAddress(recipient);
      ca.setParsedAddress(recipient);
      ca.setIMProviderType(IMProviderTypes.values()[providerId]);
      c.setContactAddresses(new ContactAddress[] { ca });

      MessageItem msgItem = null;

      msgItem = this.getMessageItem(messageId);
      c.setLastMessageItem(msgItem);

      int count = contactList.size();
      boolean found = false;
      for (int i = 0; i < count; i++) {
        ContactItem c2 = contactList.get(i);
        if (c2.getContactAddresses()[0].equals(ca)) {
          found = true;
          MessageItem m2 = c2.getLastMessageItem();
          if (m2 == null
              || c2.getLastMessageItem().getMessageId() < messageId) {
            c.setLastMessageItem(msgItem);
            contactList.set(i, c);
            break;
          }
          break;
        }
      }
      if (!found) {
        contactList.add(c);
      }
    }
  }
  recipientCursor.close();
  return contactList;

}

I’ll admit, it’s pretty inefficient and here’s why. This generates a contact list based on your database messages. It first builds a small contact list based on every received message with including the last logged message. It does another SQL search per message to get that message information. So, if you have 100 contacts, we’re at 101 searches (1 for entire recipient list, 100 for each message). This is done via the getMessageItem(messageId) function.

Then it’ll search based on sender, who you’ve sent messages to (because sometimes we have to treat somebody as a contact, even though they’ve never responded). It’ll do the same. 1 search for result, and cross reference the received message list to append or update last message info and contacts.

So, we’re looking at 1 + x + 1 + (something between 0 and x) number of SQL queries requests using this method. If you were use batch the message information requests with an “IN” clause then it’ll be 1 + x + 1. Here, ‘x’ in signifies how many contacts you have. This is all achieved using the basic android/sqlite .query() function.

There’s also the issue where just because the message was written last to the database doesn’t mean it’s the last message received. This is gets screwed up when you start syncing old messages. In other words, instead of using MessageID, we should be using CreationDateTime

So what about raw queries? Well, compare and see. Here’s the optimized implementation:

public List<ContactItem> getContactsList() {
    String rawQuery = ""
        + "SELECT "
        + "  M2.MessageID, "
        + "  M2.ProviderID, "
        + "  M2.MessageStatusID, "
        + "  M2.IsIncoming, "
        + "  M2.Sender, "
        + "  M2.Recipient, "
        + "  M2.CreationDateTime, "
        + "  M2.LastSendAttemptDateTime, "
        + "  M2.CompletionDateTime, "
        + "  M2.MessageText, "
        + "  M2.ExtraData, "
        + "  M2.ExtraDataTypeID, "
        + "  M2.ImportMessageID, "
        + "  M2.ImportConversationID, "
        + "  M2.IsRead, "
        + "  LastMessageData.ProviderID, "
        + "  LastMessageData.Address "
        + "FROM "
        + "  ( "
        + "    SELECT "
        + "      ContactAddresses.ProviderID, "
        + "      ContactAddresses.Address, "
        + "      MAX(CreationDateTime) [LastMessageTime] "
        + "    FROM "
        + "      ( "
        + "        SELECT DISTINCT "
        + "          ProviderID, "
        + "          CASE WHEN IsIncoming = 1 THEN Sender ELSE Recipient END [Address] "
        + "        FROM "
        + "          Messages "
        + "      ) as ContactAddresses "
        + "      LEFT JOIN Messages M1 ON "
        + "        M1.ProviderID = ContactAddresses.ProviderID "
        + "        AND (M1.Sender = ContactAddresses.Address "
        + "          OR M1.Recipient = ContactAddresses.Address) "
        + "    GROUP BY "
        + "      ContactAddresses.ProviderID, "
        + "      ContactAddresses.Address "
        + "  ) AS LastMessageData "
        + "  LEFT JOIN Messages M2 ON  "
        + "    M2.CreationDateTime = LastMessageData.[LastMessageTime] "
        + "    AND (M2.Sender = LastMessageData.Address "
        + "      OR M2.Recipient = LastMessageData.Address) "
        + "ORDER BY                   "
        + "  LastMessageData.ProviderID, "
        + "  LastMessageData.Address,                "
        + "  M2.MessageID";
    Cursor cursor = db.rawQuery(rawQuery, null);
    List<ContactItem> contactList = new ArrayList<ContactItem>();
    if (cursor != null) {
      while (cursor.moveToNext()) {
        ContactItem c = new ContactItem();
        MessageItem m = parseMessageItemCursor(cursor);
        int index = allFields.length;
        int providerId = cursor.getInt(index++);
        String address = cursor.getString(index++);
        c.setDisplayName(address);
        c.setDisplayContactAddress(address);
        ContactAddress ca = new ContactAddress();
        ca.setDisplayAddress(address);
        ca.setParsedAddress(address);
        ca.setIMProviderType(IMProviderTypes.values()[providerId]);
        c.setContactAddresses(new ContactAddress[] { ca });
        c.setLastMessageItem(m);

        //check for extremely rare chance of exact same time
        int count = contactList.size();
        boolean found = false;
        for (int i = 0; i < count; i++) {
          ContactItem c2 = contactList.get(i);
          if (c2.getContactAddresses()[0].equals(ca)) {
            found = true;
            break;
          }
        }
        if (!found) {
          contactList.add(c);
        }

      }
    }
    cursor.close();

    return contactList;
}

Only one SQL query. SQL will return everything I need in a neat little package. The only thing I have to worry about is messages with exactly duplicated SenderID, Address and CreationDateTime. It’s ridiculously rare for a message to have an identical timestamp, but just in case, the ORDER BY clause means when that happens, the most recently inserted message would be on the bottom. I then check if the contact already exists and update it accordingly.

With a proper index on ServiceID and Sender/Recipient, then we can generate and load contact info in less than a second even when there are THOUSANDS of messaging in the database.

Experimenting with landscape and UI changes

Don’t you hate how when typing in landscape, the entire screen turns into a message box?

Well I’ve been experimenting with different layout UIs, but take a look at what I’m trying to perfect:

landscape example

Now you can see what’s actually going on when you message. I’m considering adding some shortcut under the send button what do you guys think?

Also, you’ll see I’ve included some UI changes per suggestions and added my Settings icon. You have the following 4 options (so far) in Settings > UI > Messages

  • IM Service Indicator (ON)
  • IM Service Icon (OFF)
  • Contact Picture (ON)
  • Colored Status (ON)

Theme options all stack. I think I’m going to leave this as default except for Colored Status, but it shows how you can customize it to your liking. Also, that Dark Theme regular contact picture I had to whip up in Photoshop.

Leave your suggestions for landscape view!

Google Voice: Push limitations

I’m going to explain a couple of things about the Google Voice protocols and how push works.

As far as I know, (please correct me if I’m wrong), Google has only created 4 push implementations.

  • Google Talk message on text message/voicemail receipt
  • Email on text message/voicemail receipt
  • SMS on text message/voicemail receipt 
  • Android C2DM trigger on text message/voicemail receipt

Noticing a pattern? Yes, there is only a push implementation on new message or voicemail. There is no push implementation on:

  • Mark as read
  • Delete message
  • Archive message
  • Send outgoing message

All these items have to polled. This means they will have to run on a timer. I’m just writing this point to explain that these things ARE NOT instant and probably never will be. Only incoming communications are pushed. This means, if you’re messaging from the web interface, you might have to force a refresh or wait a minute for things to sync up on your phone.

Conversation UI (Sneak Peek)

Here’s a very preliminary UI design for messaging. It’s modelled (read: stolen) from the stock MMS XML. I’m going to change what the status text says since the full date is just a placeholder. The icons signify “Sent via” and “Received via”. Here I’m messaging myself. Take a look at how long it takes to receive a sent message. Yeah, it’s much faster than Samsung’s implementation.

Edit: just noticed Google Voice incoming has wrong date. Beta indeed

I also find showing the contact picture EVERY message is ridiculously redundant while the service icon is both informative and provides a comfortable minimum spacing for touch input. The spinner in the top left changes the destination.

Oh yeah, dark theme is beautiful. Light looks boring by comparison.

Oh and that “Settings” text will be replaced by your standard “Gear” icon but I haven’t got around to doing so. From there I’ll add the “Add” button for MMS stuff.

You can get back to contacts by pressing back, the “up/home” button on the top left or, since this is a view pager, swiping to the left.

This was posted via my phone just now.

conv

EDIT: I’ve made some more changes you can see here as per suggestions in these comments. I’m including a screenshot here since this post hit Reddit recently.

landscape example

GVoice Push Notification: FINISHED

It took forever to get a working Google Voice Push Notification system. I’ve tried using the old MASF protocol, the new JSON protocol with a custom intent, trying to intercept the official Google Voice app’s intents. None of that was working. But as of a few minutes ago, it’s done. It works. It works completely independent of whatever the Google Voice application is doing.
You don’t need the Google Voice app installed at all to get it working. It uses Android’s C2DM (Cloud to Device Messaging) framework.

Right now, there’s only one really small issue and that is: if you install Hermes after installing Google Voice, you won’t get notifications. If you install Hermes first and THEN install Google Voice, they both work. This is because both applications use the same intent permission using C2DM. It sounds a little weird but the way Android permissions work, the first app that installs a permission gets to dictate those permission’s protection level. So, when you install Hermes first, it creates a global permission stating any application can use the “Google Voice Inbox Notification C2DM” Intent (don’t worry, it’s listed as a permission, so not just any app can get that message). When you install Google Voice after, Android won’t change the permission’s protection level. If you install the Google Voice app first, it’ll lock it down to only itself, Google Voice app (boooo).  

TLDR: Uninstall Google Voice, install Hermes, install Google Voice again. 

The reality is, if you don’t use the official Google Voice app, then you don’t have any conflicts whatsoever. I’m working on a workaround to avoid having to do this uninstall. I should be able to let Google Voice get the intent and then, intercept it, after the fact. But if I can’t get it working, worst case scenario: you uninstall and re-install Google Voice.

But this was the biggest issue holding back the public beta release. Now, it’s gone. :)

More Google Voice API information

So it turns out there is a THIRD api: MASF

I have SMS out working on this “new” protocol. It’s actually the old mobile protocol based on MASF (Mobile Application Sensing Framework). It doesn’t use JSON. It uses Google’s custom ProtoBuf design. I was looking into this because I’ve run into a problem with notifications. Using the Mobile API, it seems like it’s not working at all right now. Seriously, even on the official Google Voice app my notifications have stopped working. I might have bugged my account. It should expire in a couple of hours.

For reference, the notification API works like this:

Create Register Destination that includes your account name, android ID and requested callback intent. It’s sent via C2DM.

Check in ever so often (I think official Google app does it every 60 minutes).

Create ‘Unregister Destination’ and send it when you no longer want notifications

There exists a problem that the original MASF used the correct abbreviations: Checkin = ‘in’; Register Destination = ‘rd’; Unregister Destination = ‘ud’. But apparently Google messed up when ported the MASF code and swapped ‘rd’ with ‘ud’. Now I don’t know what to trust.

For the curious, the MASF url is: https://www.google.com/m/appreq/gv

You have to POST data in ProtoBuf format. Also, the auth token required isn’t the same as “Google Voice” and uses the “grandcentral” token.

Reverse engineering is hard work.

Gearing up for public release. FEATURE REQUESTS

Well, Google Voice is looking good. I abandoned the Web API and I’m just using the highly undocumented Mobile API. Code is super clean too. So now I’m wondering what to bake in to finalize a public beta.

Also, can somebody explain to me what Group Messaging is? I’ve heard it requested but I’m not sure what it is.