                         ----------------------
                            Tuptime    Manual
                         ----------------------
                              version 5.0.2
                                Ricardo F.
                               02/Jan/2021



============
| Abstract |
============

Tuptime report historical and statistical real time of the system, keeping it between restarts.
Indeed, it can:

  - Count system startups
  - Register first boot time (since installation time)
  - Count nicely and accidentally shutdowns 
  - Uptime and downtime percentage since first boot time
  - Accumulated system uptime (running and sleeping), downtime and total
  - Register used kernels and boot IDs
  - Report current uptime
  - Print formatted table or list with the system history
  - Narrow reports since, until or at a given startup or timestamp
  - Output in csv format



=================
| Prerequisites |
=================

Operating systems:
   - Linux
   - BSD

Software:
   - Python 3.x
   - Python modules (most included in python core by default):
       sys, os, argparse, locale, platform, signal, logging, sqlite3, datetime

If a distro is used, all is met with:
    - In Linux:  'python' package >= 3.0
    - In FreeBSD: 'python3' and 'py36-sqlite3'



================
| Installation |
================

For Linux systems:

   + From distribution repository:

       Debian  - https://packages.debian.org/tuptime

       Ubuntu  - https://packages.ubuntu.com/tuptime

       Arch    - https://aur.archlinux.org/packages/tuptime/

       Fedora, CentOS  - https://copr.fedorainfracloud.org/coprs/frankcrawford/tuptime/


   + Using bash script:
      
       Download and execute the installation script:

           # bash <(curl -Ls https://git.io/tuptime-install.sh)

   + Manual install:

       Clone repository and copy executable file:

           # install -m 755 tuptime/src/tuptime /usr/bin/tuptime

       Add _tuptime user:

           # useradd --system --no-create-home --home-dir '/var/lib/tuptime' \
             --shell '/bin/false' --comment 'Tuptime execution user' _tuptime

       Execute tuptime with a privilege user for create DB path:

           # tuptime

       Change owner of the DB path and file:

           # chown -R _tuptime:_tuptime /var/lib/tuptime

       Copy cron file:

           # install -m 644 tuptime/src/cron.d/tuptime /etc/cron.d/tuptime

       If it use systemd, copy service file and enable it:

           # install -m 644 tuptime/src/systemd/tuptime.service /lib/systemd/system/tuptime.service
           # systemctl enable tuptime.service && systemctl start tuptime.service

           (Alternative) Using systemd timer instead of cron file:

               # install -m 644 tuptime/src/systemd/tuptime-cron.* /lib/systemd/system/
               # systemctl enable tuptime-cron.timer && systemctl start tuptime-cron.timer
               # rm /etc/cron.d/tuptime

       Or if it is an old RedHat system or derivate, copy the init file:

           # install -m 755 tuptime/src/init.d/redhat/tuptime /etc/init.d/tuptime
           # chkconfig --add tuptime && chkconfig tuptime on

       Or if it is and old Debian system or derivate, copy the init file:

           # install -m 755 tuptime/src/init.d/debian/tuptime /etc/init.d/tuptime
           # update-rc.d tuptime defaults
           # /etc/init.d/tuptime start

       That's all, enjoy it.


For BSDs systems:

   + On FreeBSD and derivates using packages or ports:

           # pkg install tuptime
      
           (or)
       
           # /usr/ports/sysutils/tuptime

       Add the suggested cron entry and enable it on rc:

           # sysrc tuptime_enable=YES
           # service tuptime start

   + Manual install on FreeBSD, OpenBSD, NetBSD and derivates:

       Please, read misc/bsd-manual-install.txt


Partial support for OSX (Darwin) system:

   Tuptime is compatible with OSX, but the integration with LaunchD is not
   complete. Please, read misc/osx-notes.txt


Note about Python 2.7:

   Tuptime <= 3.5.0 can be executed with Python 2.7.
   Set the shebang line to '#!/usr/bin/env python' to get it.



===========================
| Command line parameters |
===========================

These are the command line options, no configuration file is used:

  -h, --help            show this help message and exit
  -A STARTUP, --at STARTUP
                        restrict to this startup number
  -b, --bootid          show boot identifier
  -c, --csv             csv output
  -d DATETM_FORMAT, --date DATETM_FORMAT
                        datetime/timestamp format output
  -e DECIMALS, --dec DECIMALS
                        number of decimals in percentages
  -f FILE, --filedb FILE
                        database file (/var/lib/tuptime/tuptime.db)
  -g, --graceful        register a graceful shutdown
  -i, --invert          startup number in reverse count
  -k, --kernel          show kernel version
  -l, --list            enumerate system life as list
  -n, --noup            avoid update values into DB
  -o TYPE, --order TYPE
                        order enumerate by [u|r|s|e|d|k]
  -p, --power           show power states run + sleep
  -r, --reverse         reverse order in list or table output
  -s, --seconds         output time in seconds and epoch
  -S STARTUP, --since STARTUP
                        restrict from this startup number
  -t, --table           enumerate system life as table
  --tat TIMESTAMP       system status at epoch timestamp
  --tsince TIMESTAMP    restrict from this epoch timestamp
  --tuntil TIMESTAMP    restrict until this epoch timestamp
  -U STARTUP, --until STARTUP
                        restrict up until this startup number
  -v, --verbose         verbose output
  -V, --version         show version
  -x, --silent          update values into DB without output


-A <STARTUP>, --at <STARTUP>
  Restrict values only to this startup number. Take it from (-t) table or
  (-l) list reports. Negative values allowed.

  Examples:
          tuptime -A 34
          tuptime --at 22
          tuptime -l -A -1  # List last register


-b, --bootid
  Add boot identifier to the output.

  Examples:
         tuptime -b
         tuptime -t --bootid


-h, --help
  Print a quick reference of the command line parameters.

  Examples:
         tuptime -h
         tuptime --help


-c, --csv
  Report in csv format.

  Examples:
         tuptime --csv
         tuptime --csv -t


-d <DATETM_FORMAT>, --date=<DATETM_FORMAT>
  Change the datetime/timestamp format. By default the output use the
  configured system locales. Allow values are:

        %a	Weekday as locale’s abbreviated name.
        %A	Weekday as locale’s full name.
        %w	Weekday as a decimal number, where 0 is Sunday and 6 is 
        	Saturday.
        %d	Day of the month as a zero-padded decimal number.
        %b	Month as locale’s abbreviated name.
        %B	Month as locale’s full name.
        %m	Month as a zero-padded decimal number.
        %y	Year without century as a zero-padded decimal number.
        %Y	Year with century as a decimal number.
        %H	Hour (24-hour clock) as a zero-padded decimal number.
        %I	Hour (12-hour clock) as a zero-padded decimal number.
        %p	Locale’s equivalent of either AM or PM.
        %M	Minute as a zero-padded decimal number.
        %S	Second as a zero-padded decimal number.
        %f	Microsecond as a decimal number, zero-padded on the left.
        %z	UTC offset in the form +HHMM or -HHMM (empty string if the 
        	the object is naive).
        %Z	Time zone name (empty string if the object is naive).
        %j	Day of the year as a zero-padded decimal number.
        %U	Week number of the year (Sunday as the first day of the week) 
        	as a zero padded decimal number. All days in a new year 
        	preceding the first Sunday are considered to be in week 0.
        %W	Week number of the year (Monday as the first day of the week) 
        	as a decimal number. All days in a new year preceding the 
        	first Monday are considered to be in week 0.
        %c	Locale’s appropriate date and time representation.
        %x	Locale’s appropriate date representation.
        %X	Locale’s appropriate time representation.
        %%	A literal '%' character.


  Examples:
         tuptime -d '%X %x'  # (Default)
         tuptime -d '%H:%M:%S   %m-%d-%Y'  # British style
         tuptime -d '%H:%M:%S   %d-%b-%Y'  # Spanish style


-e, DECIMALS, --dec DECIMALS
  Change the decimal lenght in percentages. The number is rounded to this value.

  Examples:
         tuptime -e 7
         tuptime --dec 9


-f <FILE>, --filedb=<FILE>
  Define an alternative database file. Default is located in 
  '/var/lib/tuptime/tuptime.db'. It takes precedence over environmental variable
  'TUPTIME_DBF'.

  Examples:
         tuptime -f /var/lib/tuptime/tuptime.db  # (Default)
         tuptime -f /tmp/test1.db
         tuptime --filedb /tmp/test2.db

  
-g, --graceful
  Register the time in DB as a graceful shutdown. This option is the way that
  tuptime have for know if is a good or a bad shutdown. This is used in the 
  init.d and systemd files at the shutdown process.

  Examples:
         tuptime -g
         tuptime --graceful


-i, --invert
  Print startup number in reverse count starting since last boot. Only works in
  conjunction with (-t), (-l), or (--tat) options. Equal than the output of 
  'journalctl --list-boots'.

  Examples:
         tuptime -t -i
         tuptime -tib
         tuptime -l --invert


-k, --kernel
  Add kernel information to the output.

  Examples:
         tuptime -k
         tuptime -lk
         tuptime --kernel
         tuptime -t --kernel


-l, --list
  Enumerate system life as a list.

  Examples:
         tuptime -l
         tuptime --list


-n, --noup
  Avoid update values in the DB with the current btime, uptime, etc. Useful 
  when reporting modified DB files.

  Examples:
         tuptime -n
         tuptime --noup


-o [u|r|s|e|d|k], --order=[u|r|s|e|d|k]
  Order enumerate output from table or list by:
    <u>  uptime
    <r>  runtime
    <s>  sleep time
    <e>  end status
    <d>  downtime
    <k>  kernel
  Therefore, only works in conjunction with (-t) or (-l) options.

  Examples:
         tuptime -t -o e
         tuptime -l -o k
         tuptime -t --order u
         tuptime -l --order d


-p, --power
  Print the running and sleeping accumulated time for the correspondent uptime slot.
  Note that only Python >= 3.6 can register this values.

  Examples:
         tuptime -p
         tuptime --power
         tuptime -tp
         tuptime -l --power


-r, --reverse
  Reverse order for table or list output.

  Examples:
         tuptime -tr
         tuptime -lur
         tuptime -teo --reverse
         tuptime -l --reverse


-s, --seconds
  Change default human readable datetime/timestamp style and print times in seconds
  and datetimes in epoch.

  Examples:
         tuptime -s
         tuptime --seconds


-S <STARTUP>, --since <STARTUP>
  Restrict values only from this startup number. Take it from (-t) table or
  (-l) list reports. Negative values allowed.

  Examples:
          tuptime -S 34
          tuptime --since 22
          tuptime -l -S -10

  
-t, --table
  Enumerate system life as a table.

  Examples:
         tuptime -t
         tuptime --table


--tat
  Report system status at specific timestamp, register number enclosed and time
  elapsed and remaining in that matched status.

  Examples:
         tuptime --tat 1555280149
         tuptime --tat `date -d "1 day ago" +%s`
         tuptime -cs --tat 1555281311


--tsince <TIMESTAMP>
  Restrict report from the given epoch timestamp. Negative values allowed.
  Options (-S) and (-U) have precedence over this one.

  Examples:
         tuptime --tsince 1454998872
         tuptime --tsince `date -d "20-JAN-18" +%s`
         tuptime --tsince -31536000  # Since one year ago


--tuntil <TIMESTAMP>
  Restrict report until the given epoch timestamp. Negative values allowed.
  Options (-S) and (-U) have precedence over this one.

  Examples:
         tuptime --tuntil 1454999619
         tuptime --tuntil `date -d "2018-01-20 16:21:42" +%s`
         tuptime --tuntil -2592000  # Until one month ago


-U <STARTUP>, --until <STARTUP>
  Restrict values only up until this startup number. Take it from (-t) table or
  (-l) list reports. Negative values allowed.

  Examples:
          tuptime -U 45
          tuptime --until 11
          tuptime -l -U -10


-v, --verbose
  Print information about the internals of tuptime. It's good for debugging 
  how it gets the variables.

  Examples:
         tuptime -v
         tuptime --verbose


-V, --version:
  Print version number and exit.

  Examples:
         tuptime -V
         tuptime --version


-x, --silent
  Any output is printed. Only update values and save them into disk.

  Examples:
         tuptime -x
         tuptime --silent



=========================
| Environment variables |
=========================

List of supported environmental variables:

TUPTIME_DBF
  Set an alternative database file path. The argument -f, --filedb takes
  precedence over this.

  Example for ksh / sh / bash:
         export TUPTIME_DBF="/opt/tuptime.db"

  Example for csh / tcsh:
         setenv TUPTIME_DBF /opt/tuptime.db 

  Example for systemd unit:
         Environment="TUPTIME_DBF='/opt/tuptime.db'"



======================
| Schedule execution |
======================

It's important to have a schedule execution. If the init or systemd scripts
are installed as it needs, the program only update values at startup and 
shutdown, but if the system fails, hangs or whatever, the uptime time will
be lost. Be sure that the cron entry or, in other case, the systemd
tuptime-cron.[timer|service] units are installed as expected.

Why it's needed the use of an other .service file with the name tuptime-cron?
The default tuptime.service file uses the option 'RemainAfterExit=true' and
nowadays the systemd timer can't restart an already running service, so, this
is the best workaround.

By default, both have a 5 minutes scheduled execution. Feel free to lower 
that value if your requirements are narrow.



==================
| Default output |
==================

System startups:
 Total number of system startups registered since first timestamp available.

System shutdowns:
  Total number of shutdowns done correctly or incorrectly.

System life:
  Time counter since first startup timestamp available.

System uptime:
System downtime:
  Percentage of time and time counter.

Average uptime:
Average downtime: 
  Time counter with the average time.

Current uptime:
  Actual time counter and datetime since registered boot timestamp.


Removed on version 5 and equivalent output:

  Largest uptime:      tuptime -tor u | head -3
  Shortest uptime:     tuptime -to u | head -3
  Largest downtime:    tuptime -tor d | head -3
  Shortest downtime:   tuptime -to d | head -3



======================================
| DB format changes between versions |
======================================

From 2.x to 3.x:
    From 2.x to 3.0, reorder columns, added offbtime, endst and downtime. From 3.0 to 3.1,
    added kernel column. The migration scripts are loated at:
    
    https://github.com/rfrail3/tuptime/blob/master/misc/cripts/db-tuptime-migrate-2.0-to-3.0.sh
    https://github.com/rfrail3/tuptime/blob/master/misc/cripts/db-tuptime-migrate-3.0-to-3.1.sh
 
From 3.x to 4.x:
    Added rntime and slptime registers, also uptime, rntime, slptime, downtime are now integers.
    The migrations is done automatically from Tuptime. Also, the migration script
    is located at:

    https://github.com/rfrail3/tuptime/blob/master/misc/cripts/db-tuptime-migrate-3.1-to-4.0.sh
 
From 4.x to 5.x:
    Added bootid register. The migrations is done automatically from Tuptime.
    Also, the migration script is located at:

    https://github.com/rfrail3/tuptime/blob/master/misc/cripts/db-tuptime-migrate-4.0-to-5.0.sh


==================
| Database Specs |
==================

Tuptime use a sqlite3 database located in '/var/lib/tuptime/tuptime.db' with
the following format:

tuptime (bootid text,
         btime integer,
         uptime integer,
         rntime integer,
         slptime integer,
         offbtime integer,
         endst integer,
         downtime integer,
         kernel text)

bootid   Unique boot identifier (if it exists)
btime    Startup timestamp in epoch
uptime   Uptime seconds  
rntime   Running time seconds
slptime  Sleeping time seconds
offbtime Shutdown timestamp in epoch
endst    Type of shutdown [1 ok | 0 bad]
downtime Downtime seconds
kernel   Name of the kernel

The number of startup sequency is extracted from 'rowid', the signed integer 
key that uniquely identifies the row within its table in sqlite.

For reset all the values, simply delete the database file.

It is possible to query and modify it directly, but it's better to use any of
the scripts located under 'misc/scripts' for modify, join or check it.

If a complete row is deleted and the database is not recreate completely (vacuum),
rowid keep the real information about startup number. Tuptime will advert about
it if the verbose mode is enabled.



============================
| About sync date and time |
============================

Large jumps over the date and time in a system can cause problems, expecially
when the uptime value is low.

Take care of the harware clock RTC (if the system have), or how the operating
system initializes their clock at startup. Tuptime register the timestamp
reported by the system at first execution after boot and use it to calculate
the previous shutdown time and the current uptime value. The command 'who -b'
do more less the same.

If the system starts with a incorrect datetime and some external time sync is
used, like ntpd or timesyncd, a few time after boot the startup datetime reported
by the system may change. This behaviour is also related with adjustments
performed by adjtime(3), systems running inside virtualized enviroments, servers
with high load or with high disk I/O, wrong computation of jiffies / HZ and the
problema of lost ticks.

A slightly drift is compensate over the uptime value, but a very high drift can cause
abnormal registers and output, for example, Raspberry Pi don't have any clock and
initializes from 1 January 1970. A downtime of 0, when it's clear than it was
physically impossible, it's a common indicator than the clock had a large backward
sync jump.

As Tuptime have time dependency, their Systemd service unit pulls time-sync.target
and it's ordered before it. Note that it doesn't assure that the clock is syncronized,
only that it needs it. On last releases of Systemd, there is a new unit that delays
reaching time-sync.target until stay on sync 'systemd-time-wait-sync',but it isn't
enable by default nowadays.

A few recomendations / workarounds are:

Enable 'systemd-time-wait-sync'. But take care because it can delay the execution
more than expected and a startup can be lost. Check open issues on Systemd's GitHub.

Use 'systemd-timesyncd' if it is available (timedatectl set-ntp true), it can also
keep the time in systems without hardware clock. Disable any other time sync service
like 'ntp', 'chrony', 'openntpd' to avoid issues, most related with the use in
conjuntion with 'systemd-time-wait-sync'.

Use any time sync service like 'ntp', 'chrony' or 'openntpd' and disable
'systemd-timesyncd'. Add the time sync service requirement on 'tuptime.service' at
the end of 'After=' and 'Wants=' lines.

Force a time sync in the 'tuptime.unit' file. Add under '[Service]' definition the 
line 'ExecStartPre=/usr/bin/ntpdate pool.ntp.org'

If the systems is a single-board computer, like Raspberry Pi or Beaglebone, use a
RTC module and configure it properly.

Without Systemd, use a ntp client and delay the execution of tuptime until the time
is synced. Also, 'ntp-wait' can be added to the init script and the cron line
preceeding the tuptime execution.

Try and failure is the best way to find the right solution for the environment in 
which the system is running.

The drift value that Tuptime have reference is reported in verbose mode 'tuptime -v'
in the line 'INFO:Drift over btime'. Values around +-1 are common and can be
considered as normal.

Systemd timer issues:

    https://github.com/systemd/systemd/issues/14061
    https://github.com/systemd/systemd/issues/8683
    https://github.com/systemd/systemd/issues/5097

Good reads:

    https://tools.ietf.org/html/rfc1589
    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=119971
    http://man7.org/linux/man-pages/man2/clock_gettime.2.html
    https://unix.stackexchange.com/questions/118631/how-can-i-measure-and-prevent-clock-drift
    Timekeeping in VMware Virtual Machines (Document)
    https://venam.nixers.net/blog/unix/2020/05/02/time-on-unix.html



========================
| Date and Epoch notes |
========================

The options --tsince, --tuntil and --tat use epoch timestamp format, as quick
reference here are some notes:

Seconds equivalences:
        1 year    31536000 seconds
        1 month    2592000 seconds
        1 week      604800 seconds
        1 day        86400 seconds
        1 hour        3600 seconds

Convert human datetime to epoch:
        date -d "JAN-18-2018" +%s
        date -d "20-JAN-18" +%s
        date -d "2018-01-20 16:21:42" +%s
        date -d "2018-01-20 16:21:42 GMT+2" +%s
        date -d "3 week ago" +%s
        date -d "1 year ago" +%s
        date -d "now" +%s

Convert epoch to human datetime:
        date -d "@1516748400"

Preceding conversions uses GNU date, on FreeBSD install "coreutils" and call
them with "gdate".

More info about datetime input formats:
        man date
        'https://www.gnu.org/software/coreutils/manual/html_node/
        Date-input-formats.html#Date-input-formats'



===========
| Caveats |
===========

Tuptime can be executed with 'root' user, but it's recommendable the use of an
unprivileged user, like '_tuptime' as the installation section points to avoid
security problems. In any case, if the database file is not writable by the
current execution user, the changes are not saved. This situation is covered
for print values, but not for update database, that's why a user who can write
in the database file must execute tuptime, usually from the init manager, to
update values at startup and shutdown, at least.

Kernel name is registered each time that tuptime update values. If ksplice is
used to update the kernel, last kernel will have the whole time assignment
since the last system start.

Tuptime is backed by a SQLite dabase. It is highly resilient but fails can
occur in any software and hardware level, producing a database corruption. Here
is an excellent documentation about it https://www.sqlite.org/howtocorrupt.html

The uptime reference is linked with the wall clock time elapsed since boot, not
the running efective time. Running (efective time) and hibernate power states
are childs of the uptime and report accumulated values inside their time range,
which counts since the startup until the shutdown.

If the system have configured a time zone with daylight saving time (DST),
to avoid problems when the time change, the real time clock (RTC) should have
set to universal time coordinated (UTC), not local time. Check it with 
'timedatectl' command.  Reports involving DST changes are completely right.
The system current local time zone is the reference for all timestamps printed.
https://tldp.org/HOWTO/Clock-2.html

The name of the tool is based on the contraction of the words Total Uptime.
