Here’s something I’ve been working on for the last month or so…
This is a simple life logging script designed to work with the excellent Day One for MacOS X. While I use it in conjunction with IFTTT and Dropbox, there’s nothing in the code that requires this setup. It would work just as well with Google Drive, for example, or you could even pull data down yourself from various RSS feeds and drop the data into a local directory in the right format.
Old data is never directly deleted, but instead archived in a location of your choice (which may very well be the Trash…).
lifelog.rb was developed under Ruby 2.1.1; it may very well run with early versions, but I haven’t tested it. The only non-standard gem used is
lifelog.rb also makes a call out to the
terminal-notifier binary to produce notifications.
I recommend installing Ruby using
ruby-build; both of those, along with the
terminal-notifier binaries, can be installed using Homebrew. Check out the rbenv GitHub page for details on how to get Ruby working.
Once you’ve got Ruby 2.1.1 installed, just use
gem install exiftool to pull down the
lifelog.rb is configured via a simple YAML file at
~/.lifelog.yml an almost-ready-to-use version of this file can be found in this repository with the name
lifelog.yml. It contains the following configuration options, all of which are currently mandatory:
lifelog_dir: A string defining the raw log storage directory (see Providing Data, below).
archive_dir: A string defining the directory old log files will be archived in after processing.
log_order: An array of strings defining the order in which different logs will appear (see Providing Data, below). Logs not defined here will still be processed, but will appear in alphabetical order after those on this list.
day_boundary: A (military) time (
%H:%M, i.e., 00:00 - 23:59) denoting when a “logical” day should end. Set this to a time you are certain to be asleep. Events before this time will be considered part of the previous day for logging purposes (hence setting it to 00:00 will cause logged days to align with calendar days). You must quote this string, as otherwise Ruby’s YAML gem will interpret the colon in weird ways.
day_one: A list of Day One specific options:
journal_dir: The location of your Day One Journal (i.e., the
starred: Should life log entries be automatically starred in Day One? I can’t think of a reason anyone would want this, but it’s included for completeness.
tags: An array of tags that new life logs entries should be automatically assigned. Useful for distinguishing automatically created entries from things you write yourself.
lock_file: By default,
lifelog.rbwill only run once a day, and will not process entries from the current “logical” day. This is done to prevent incomplete journals from being entered into Day One. The
lock_filewill have the current day’s timestamp written out to it, as a way to quickly prevent the rest of the script from executing if the time is not yet ripe.
lifelog_dir is expected to have the following structure:
./Day One/weather.txt ./Day One/Photos/*.jpg ./Log1/log.txt ./Log1/Atomic/*.txt ./Log2/log.txt ./Log2/Atomic/*.txt ... etc. ...
Day One directory is special, and is composed of items that will be integrated into the Day One journal entry but are not actually logs; currently this information is weather and (potential) header photos. All other directories are considered to be logs, and are expected to contain a
log.txt file of timestamped log entries and/or an
Atomic directory of non-timestamped log entries.
It is thus not possible to have a log called “Day One”. However, any other log directories should be fine.
Any directory may be missing and/or empty, in which case no log entries will be generated from that source. All file timestamps are should be in the form “April 29, 2014 at 06:08PM” in files, and
April_29__2014_at_0608PM for files.
Which is kind of a dumb format for these things, but won’t get changed until IFTTT provides better timestamp support (right now, that’s how the timestamps it generates are constructed).
Day One Directory
Day One directory contains files that represent metadata that will be written into Day One journal entries, but are not actually entries themselves. Right now this consists of a file
weather.txt recording weather conditions and a
Photos directory containing potential journal header files.
This file records weather conditions, one entry per line, in the following format:
TimeStamp :: Condition1(Value1),Condition2(Val),etc.
TimeStamp is in the format
%B %d, %Y at %I:%M%p (i.e., “April 29, 2014 at 06:08PM”) and conditions are taken from the following list:
Celsius: An integer representing the temperature in degrees Celsius.
Description: An arbitrary string (well, no parentheses) describing the weather.
Fahrenheit: An integer representing the temperature in degrees Fahrenheit.
Pressure MB: A float representing the pressure in millibars.
Relative Humidity: An integer representing the relative humidity (not sure what the units are here).
Service: An arbitrary string (well, no parentheses) indicating the service used to pull the weather data.
Visibility KM: A float representing the visibility in kilometers.
Wind Bearing: An integer representing the wind bearing (I believe in degrees as measured clock-wise from North).
Wind Speed KPH: A float representing the wind speed in kilometers per hour.
IconName: A PNG icon name. Full URLs will be accepted, but all path information will be stripped before insertion.
These conditions can occur in any order; conditions not on this list will be silently ignored. Thus, a valid entry in
weather.txt looks something like the following.
April 30, 2014 at 02:57PM :: Celsius(11),Description(Partly Cloudy/Wind),Fahrenheit(52),Relative Humidity(32),Wind Speed KPH(42),IconName(http://ifttt.com/images/weather/wind.png),Service(IFTTT)
Only one weather entry is recorded per logical day. If multiple entries are present, the latest entry for that day is used.
A directory of JPG photos, named as
TimeStamp is of the form
String is an arbitrary string. Thus, a valid photo name will look something like:
Images are first groups by date:
lifelog.rb will first attempt to use the EXIF “Date/Time Original” field, then the EXIF “Date Created” field, and finally parse the file’s leading
TimeStamp. Within a logical day, images are sorted widest to tallest, largest to smallest, and latest to earliest. Only the “first” image in the sorting (i.e., the widest image that has the highest resolution and latest timestamp of all other images with the same aspect ratio) will be used.
If EXIF GPS coordinates are present in any of these images, they will be used for the journal entry. The fall-back order for GPS coordinates is the same as for the images themselves.
All directories other than
Day One are considered to be logs, each of which is given the same name as the directory itself (so if you want a log called “Fluffy Things”, you should have a directory called
Fluffy Things). The best way to record information for each log is in a
log.txt file located within this directory. As a fall-back, log entries can also be stored in the
Atomic subdirectory, which records one entry per file. The difference between these methods is that
log.txt contains leading timestamps, whereas entries in the
Atmoic subdirectory are expected to not have timestamps (the entry timestamps will be taken from the file modification time).
log.txt contrains one log entry per row in the format
TimeStamp :: Entry, where
TimeStamp is in the format
%B %d, %Y at %I:%M%p (i.e., “April 29, 2014 at 06:08PM”) and
Entry is an (almost) arbitrary string. Since each line is considered a log entry, no line breaks are permitted within
\r strings will be replaced by newlines in the final output, so if you want to get extra line breaks you can use these. No other processing is performed on
Note that Day One stores entries internally using Markdown, so there’s a lot of formatting potential here. In my testing, it also accepts a limited number of HTML tags (in particular,
<sub/>). All strings are considered UTF-8 encoded, which means that you also get a lot of special characters to play with.
It is also possible to start off log entries with a
%B %d, %Y (i.e., “April 29, 2014”). These entries are considered “all day” events, and will be presented before all other entries in a given log. “All day” events are separated from regular entries with a single horizontal rule.
Log entries will be sorted by timestamp, and thus do not need to appear sequentially within a log file. An example log snippet is presented below.
April 14, 2014 at 06:22AM :: Sunrise April 14, 2014 at 07:34PM :: Sunset April 14, 2014 at 02:22AM :: Went to bed April 14, 2014 at 06:45AM :: Woke up [slept 4h 14m]\r\r![Light Sleep: 66.27%, Deep Sleep: 25.29%, Awake: 8.44%](https://my.domain/image.png)\r April 14, 2014 at 08:14AM :: Black Tea, Granola, and Vanilla Non-Fat Yogurt April 14, 2014 at 09:30AM :: Meeting [[calendar event](https://my.domain/calendar/link1)] April 14, 2014 at 11:00AM :: Call [[calendar event](https://my.domain/calendar/link2)] April 14, 2014 at 01:00PM :: Training [[calendar event](https://my.domain/calendar/link1)] April 14, 2014 :: Walked 1.2 miles April 14, 2014 at 07:42PM :: Black Beans, Hot Organic Salsa, Mild Cheddar Cheese, Tortilla Chips, and Water
Sometimes you can’t get event timestamps for some reason (for example, IFTTT doesn’t offer the option to record when you archived an entry from the Pocket read-it-later service, only when the entry was first saved). In these cases, you can write out “atomic” log entries into the
Atomic sub directory. Each entry should be a single file containing a single line with the same formating as the
Entry string described in the previous section. In this case, entry timestamps will be computed using the log file modification time.
This is obviously less-than-ideal. You should always record entries in
log.txt when possible.
lifelog.rb will begin checking to see if the
lock_file contains today’s date. If it does, then the script exists. Otherwise it creates a directory named
TimeStamp is of the form
The script then descends into the
Day One directory, moving
weather.txt to a corresponding directory in today’s archive and then parsing out all weather events. Any events that occur in the current logical day (or in the future) are written back to a new
Day One/Photos directory is then moved into today’s archive, and all photos are processed with
exiftool. Photos corresponding to the current logical day or later are copied back to their original location.
Finally the log directories are processed. Each log directory is moved into today’s archive, and then
log.txt is processed, with events from the current logical day or later copied back to a
log.txt file in the original location. The
Atomic subdirectory is then moved as well, and every file within it is processed. As before, files corresponding to the current logical day or later are copied back to their original location.
The end result of this is that all data for past logical days is read in and archived, while data for the current logical day and any future days is returned to its original location and not read in.
Once all data is read in, Day One journal entries for every logical day since the date recorded in the
lock_file are generated (recall that no data is read in for the current logical day or any later days). If photos exist for a particular day, then the appropriate one is copied into the Day One journal for that day.
Once all journal entries are written out, today’s date is written out to
lock_file and the script exits.
lifelog.rb is the first substantial script I’ve written in Ruby, and I’m sure it has problems. Here’s the ones I currently know about:
Sensible defaults should be set for configuration options not included in
There are definately file encoding issues. Right now I force all text to be interpreted as UTF-8, but I really should try to detect what my encoding is instead (and probably convert to UTF-8).
At the moment I do no time zone determination. In fact, I explicitly throw all time zone information away. But time zone support is really a must. (This will need to wait until IFTTT implements better timestamp functionality.)
Speaking of timestamps, it would be nice to let users choose their own format. (This will need to wait until IFTTT implements better timestamp functionality.)
Speaking of time zone support, once we have it we should note time zone transitions in the journal.
At the moment I make system calls to determine current hardware and OS versions. It would be good to replace this with pure Ruby code.
Geolocation is essentially unimplemented, except for pulling latitude and longitude out of header images (if that information exists). It would be good to add in better location support, including inserting a default location if no header image exists. However, none of the Ruby geolocation gems I tried seemed to work reliably.
Calling out to
exiftoolto get image metadata seems… Broken. Better to use the Ruby bindings for
exiv2, perhaps. However, none of the projects providing
exiv2bindings as a gem seem to provide adequate functionality or be actively maintained.
The Day One Mac app displays images correctly rotated based on the embedded EXIF rotation data, but the iOS app does not. While this looks to be an issue with Day One on iOS, it would probably be worth just rotating all images to match their embedded rotation data. It looks like this can be done using the
auto_orient!method of the RMagick gem.
The current use of
day_boundaryto arbitrarily provide a cut-off for today seems too coarse. However, simply relying on sleep times results in too many edge cases if one is inclined to take naps. A better solution is needed.
Right now entry time is taken to be the last journal entry of the logical day, or 23:59:59. It would be nice to consider the time stamp of header photos as well, though as a practical matter I think this will rarely result in a different computed time.
At the moment journal entries that correspond to previously processed days are silently ignored. It would be better to append these entries to the corresponding log in the appropriate journal. This is a lot harder to do than it sounds.
I can’t figure out how to get
terminal-notifierto use Day One’s icon instead of it’s own. Help!
Bug reports, ideas, and contributions welcome. As I noted previously, I use this script every day, so I’m not going to accept a change that breaks my usage or adds undo complexity. But I’m sure there’s still a lot of room for improvement here, and would love some help making this script even better (and more useful for others!).