New stuff - Password Reset, User Activity Tracking

Jun 5, 2014 at 9:23 PM
Edited Jun 5, 2014 at 9:24 PM
Hi folks
A few comments for a new code push. There are two new big things in Vita.Modules assembly -
  1. Extended Login module - now includes support for multi-factor login, fully implemented password reset functionality, and automatic log recording all login-related activity.
  2. UserActivityTracking module - to support structured records of transactions and sessions initiated by particular users.
There are also a couple of half-baked modules - Party and AddressPhone - these are work in progress, but already usable to some extent.
With these new pieces we finally starting to see how module-based development in VITA is supposed to work. You now have several modules with quite substantial functionality which can be easily imported into your application. Just a few lines in your entity app class - and you have a bunch of extra tables with related functionality ready to use in your application. See Books sample - we import ErrorLog, Login, UserActivityTracking and WebCallLog modules, and a dozen of tables appear in Books database, all hooked and connected properly. What's more exciting, some of the modules allow extension of other modules - UserActivityTracking module allows adding tracking columns to tables in other modules (like Books, BookReview) to track created/updated-by(user)/on(date) properties. More on this below

Login module enhancements

Login module now uses 6(!) tables, instead of just 1, to implement the new important features. One is multi-factor authentication, allowing use of additional factor - like something you own - your phone, that will be used to send SMS message containing PIN for second step verification. The other feature is Password reset process, which is not so simple as it might seem. Troy Hunt wrote an excellent post on the topic, outlining the process in details: . It gives an excellent explanation on what you should and should not do when implementing the password reset. Login module provides the exact implementation of the process described, which can be easily imported into any real-world app. This includes support for second factor, and Secret Questions /Answers support, completed with initial list of secret questions (collected from the web) that you can easily import into your database. Unit tests for books sample include LoginTest file which demonstrates the full process - go through it line by line for more details. Secret questions are automatically imported from the accompanying text file into the books database. The system also supports custom questions that user can create for himself if he's not satisfied with the provided list. Notice that we have followed some good security practices, so users' answers to security questions are not stored in plain text, but as hashes (weak, short hashes in fact), just like other sensitive information (passwords are hashed using super-strong hashing).
The password reset and multi-factor authentication are in fact mini-workflows, spread in space and time, and a dedicated table is supporting these flows. Additionally, as Troy points out, detailed logging for security sensitive activities like Password Reset is extremely important, so Login module provides this as well - all automatic. Again, see TestLogin sample for details. After executing the tests, look at the tables data in SMS, see what had been recorded.

User Activity Tracking module

A new module is aimed to give a structured and robust solution for very common pattern - tracking created/last-updated by/ondate attributes on data records. You probably had seen or did this many times: all these "CreatedByUser", "UpdateByUser", "LastUpdatedBy", etc extra columns in database tables. Adding and maintaining these fields is an extra headache, maybe not big one, but still an extra code and need to remember to always put proper values into these fields when creating/updating data. Another problem - these columns can add substantially to disk space required for storing the data. The user column is usually at least 30, more common 50 chars long, so only createdByUser, updatedByUser columns might take an extra 100 unicode characters (200 bytes) to each record. Plus 16 bytes or so for two dates.
The suggested solution is to normalize out these fields into two tables: UserSession and UserTransaction. UserSession more-less corresponds to web/login session, the activity from user loging until logout. It contains user name, just one instance for all records touched by the user in time of the session. UserTransaction is a record automatically created by session.SaveChanges call - it records a date-time of the transaction and has a reference to parent UserSession record. The data tables now should have a reference to this transaction record like CreatedInId, UpdatedInId columns (type Guid). These references are maintained automatically, so you don't have to do anything when you submit and modify data in your business logic code - the framework maintains these fields.
There are two ways to add CreatedInId, UpdatedInId columns - explicitly declare them on an entity, or add them programmatically at app-assembly time. With explicit declaration you add a Guid property to an entity and decorate it with Track attribute:
  Guid CreatedInId {get;set;}
  Guid UpdatedInId {get;set;}
These column values are then handled by the framework automatically.
The second method is much cooler - you can add tracking columns to entities defined in other modules at the time of assembling application from modules - in the EntityApp class. You just call AddTracking method providing it with types of entities to track, and the names of extra created/updated columns. The columns would appear in database tables, but obviously you cannot access them through entity interface in c# code. But this is in fact rarely if ever needed - in investigation scenario you'd look directly at tables in the database. So you might decide to use a module designed by somebody else, while enhancing its data with tracking fields. See BooksEntityApp.cs file for an example.
Notice that we add Guid columns, not direct references to IUserTransaction entity. The result is that there will be no referential integrity contraint in the database. It's bad, but there are reasons for this. First, performance. Creating IUserTransaction record is not done in the same transaction that is doing SaveChanges call in the mainstream code; instead, it pushed to the background thread which inserts multiple accumulated records (from all executing threads) in one batch call once a second. Therefore, when we save the data the corresponding transaction record might not be there (yet), so the referential integrity link (if it was there) would reject the transaction. The other reason is the possibility that the UserTransaction table would be regularly purged (removing or copying somewhere else the too old data), to save the disk space in the main database, with reliance on backup for investigation of particular past transaction - if we ever need to do this. With strict ref integrity link the data records that had been updated long time ago would have an orphan link, which would not allow purging the data.
One benefit of this normalized tracking info in separate tables is that it is possible now to get the set of all records updated in particular transaction (if they were not overwritten yet) - you just find the transaction ID in all involved tables. This was much harder, if even possible, with traditional arrangement of individual columns in each table.
The idea of activity tracking might be brought to a new, higher level, and you probably already have a question about this - what if I need to track not only last update user/date - which is overwritten by later updates - but full history of all updates, including the old/new values updated in certain critical tables. This is a valid question, and in certain cases it is a requirement to have such a detailed log. One example is fraud prevention in financial systems - any changes to 'rates', fees table must be recorded, including who, when did this, and what values had been changed. The other example is general requirements to systems in certain areas like HIPAA (our case here), when it just says: all changes to important data must be logged, with details. Period. This is what is usually called Audit - or let's just use this term. This wil be a separate, new module, which is coming, and this is not a 'maybe' promise - I have no choice, we have to do this, it is a strict external requirement - so expect it soon.
Jun 5, 2014 at 9:28 PM
Edited Jun 6, 2014 at 3:44 AM

Some general observations.

This is kind of a big thing. With these two modules, VITA's promise of component-based development in data-connected application finally starts to show some results. These are two standard modules, pre-developed and supplied with framework, which cover a substantial functionality which is quite common, and not so easy to implement from scratch. You can easily integrate these modules into your app, and have tables appear in the database, and supporting .NET services ready to use. Think about this. There are many more common pieces that we can pre-implement and then start assembling apps from them, with tens or even hundreds of tables coming from pre-built modules - joining your database and the solution. One big piece is missing in the picture - some UI construction and 'import' facilities. We can import now database objects and supporting .NET code. What if some modules (like AddressPhone in Vita.Modules) had some default UI arrangements, which you could easily import into your page(s), with look and feel of your page's style sheets? This is definitely coming, and I'm thinking about this all the time. Any ideas?

Other improvements.

WebLog functionality and WebCallInfo object now let you specify cookies to add/remove, directly from the controller code. This is a convenience feature, just like specifying custom HTTP status code for the response. There is also an extra flag HasSensitiveData in WebCallInfo object that signals to logging system that this request contains some secret data, so it should NOT log the request's body. This should be used to prevent password leaking through logs. Login module does not save passwords, it saves strong hashes, so malicious hacker would not gain much if he gets access to logins table. But if we are not carefull, log tables might have passwords in plain text logged in incoming requests. To prevent this from happening, you should set this flag in your LoginController.Login method - and log system will not log the request Json object (it will put (Data omitted) message instead). By the way, authorization headers in HTTP request (used in Basic HTTP Authentication) are not logged ever, with or without this flag.
You can also see some another pattern emerging for modules extensibility - using generic entity module classes. Look in UserActivityTrackingModule.cs as an example. The module defines 2 entities (interfaces) representing 2 tables. The generic module definition is defined as a class with generic arguments representing these entity interfaces or something derived from them. So you can extend the definition of IUserTransaction entity, add a few custom properties, and then instantiate the module using your custom transaction entity as an argument. As a result, the application (and code inside the module) will be using your extended definition, and the table in the database will have the extra columns. The activity tracking module also contains non-generic definition (derived from generic with standard entities as argements) - you'll be using this most of the time.
This pattern of using generic module classes is an alternative to the old method of using ReplaceEntity method for customizing modules. I did not play with this extensively yet, so not sure how well this will work in practice, but the approach seems quite promising.

Finally, a couple of common 'data' modules - Party, AddressPhone. Both are first drafts and a work in progress. Party module implements a standard pattern of Pary/Org/Person objects (google it). AddressPhone is an attempt to define common set of entities for holding addresses (US addresses mostly); most likely will be revised, with Street/City/State normalized further and some other enhancements. MonitoringModule and IncidentLog - usable, but in early stage of development.
Jun 9, 2014 at 8:06 PM
Latest addition - I refactored the link between Modules and Areas - now entity module can place its entities into Areas (schemas) other than its own, default Area. LoginModule in Books sample now places LoginLog and LoginWorkflow tables into "log" schema/area in the database, where it better fits with other log-like tables.