EF 7 - your insights

Aug 8, 2014 at 6:55 AM
Hi!

I for now decided not to use vita, as it is missing too many features I am relying on (e.g. OData, ASP.NET Identity 2 Integration, SQLite support).
Another reason is that I am using sync framework and therefore I need to filter out denied records at database level.

However, I really like your approach of having entities being just interfaces and including permission control in your ORM. That is really missing in EF.
The only disadvantage of your current approach is that your datafilters do not filter on database level, but in memory. (at least that is what your dev-documentation says at the bottom of the page in the recap section)

So why am I writing this?
I want to encourage you to contribute to EF, as EF7 will involve a huge refactoring and target many of the downsides that you mentioned and criticized, like performance improvemements as they also want to target portable devices (phones, tablets) and also SQLite support.
Although the way they go seems to be quite good, I think they could really need you in their team. I also started a discussion on their github repo, that they should add a similar approach as the one you are using.

And finally I wanted to say thank you for giving me such detailed insight in authorization. :)
Coordinator
Aug 8, 2014 at 6:51 PM
Well, good luck! Sincerely, wish you the best of luck.
I'm just afraid that some day a year from now or so, you gonna start cursing the day you made this decision - to go with EF. Not the decision to pass VITA, that's understandable, but going with EF - well, good luck!
I will later respond to other items in your post (sync fwk, contributing to EF, etc)
Just one warning for now - don't buy at face value all the promises and BS about EF and other crap from Redmond. I kinda have impression that you do in some cases. So many of these cool sounding things and 'supports this and that' result in unmanageable mess in the real world. Try everything yourself, on real-sized data, give a real time and effort to bang these things hard.
Roman
Coordinator
Aug 9, 2014 at 6:17 AM
continuing.... Again, not trying to change your mind, just commenting on things you mention - helps me for sure, and it might help you even with EF, I guess.
I've looked (again) at sync framework - it does not seem to be tied to EF, ADO.NET is all it needs. And it relies on ChangeTracking functionality in MS SQL Server. So I don't see how it's connected.
Filtering data in database for authorization rules. Hm, you use quite strange logic. VITA provides authorization filtering only on the client (in c#), while EF does not provide anything. Then you say: I do need authorization filtering, preferably in database, therefore EF is a better choice. Really?!
A bit more about authorization and filtering in the database. Probably my note about search queries is somewhat confusing. Here's some clarification. Authorization in general is NOT supposed to replace the business logic, and filtering/querying as it should be. In a well working application, AccessDenied exception should never be thrown - everything works as expected, proper records are delivered using appropriate queries (user sees only his purchase orders, or only in his department). Authorization quietly works in the background and never interferes. What's the point then? It provides a solid guarantee that if you have a bug or faulty logic, or somebody is trying to trick the application (hand-crafting the URL), then Authorization will jump in and stop the operation. As such, authorization code and logic should not be 'shared' with business logic, so that it can INDEPENDENTLY verify the access rules in the background. So when it comes to filtering in the database, your queries should ALWAYS include appropriate clauses that select only allowed documents. Once the query delivers them, the authorization code will independently check that each entity/data is OK to see by the current user. Search queries are no different - even if user enters some custom criteria, after translating it into SQL you should explicitly add the clauses that limit the resultset to only allowed records. Just imagine an app without VITA's authorization, let's say EF-based app. You have to manually craft the query that respects the access rights. What VITA authorization would add to this is an extra independent check, after records had been delivered.
What would be nice (and how you should read this end note in Authorization doc) is ability to have some help from authorization subsystem in forming these queries. This is something of extra convenience, nice service to have, but its absence does not invalidate any gains you have already from auth system.
Aug 11, 2014 at 9:23 AM
Edited Aug 11, 2014 at 11:55 AM
Hi!
I see you are reading everything quite carefully. ;-)
Seems I did not explain clearly what I am doing:
Yes, Sync Framework is not tied to EF at all. However, using it, requires shifting all (at least "read") permission logic to the DB layer, as this is the interface for Sync Framework. And you are right again when you say that EF does not provide anything useful in this case.

The main reasons why I will not use VITA are therefore:
  • Is maintained by only one person - No offense, but I have seen so many open source projects been deserted and seriously - you don't get paid for maintaining VITA, right?
  • Does not plug into several technologies we will use (Breeze.js, OData, Asp.Net Identity 2, SQLite)
As we are in a very strict timeline and do not have the resources (time is money) to craft all these interfaces ourselves, I will have to go the "Mainstream Way", despite all of your (good and well explained) warnings. :/

In an ideal world, you'd be the chief designer of EF vNext so you could really craft a better EF for everyone.
Aug 12, 2014 at 2:46 PM
Just one short hint:
I HATE EF for not supporting global filters. Does VITA you support them? (see https://github.com/jbogard/EntityFramework.Filters for someone who tried to "somewhat" create this feature for EF 6)
Coordinator
Aug 16, 2014 at 6:55 AM
I'm on vacation now, in far away forests in Montana. I will reply when I'm back next week.
Coordinator
Aug 21, 2014 at 7:08 AM
Edited Aug 21, 2014 at 7:13 AM
Hi
About 'global filters'. The short answer is NO, VITA does not support global filters. But it's really easy to hack it in, - at least for LINQ queries. Just extend EntitySession, override EntitySet<TEntity>() method, and add WHERE clause as appropriate (for entity type) to IQueryable returned by base method. EntitySet<T>() is used in LINQ queries as representation of 'tables', so any query through the session against a table would include extra WHERE clause.

About this filtering in database business. Again, I think you're a bit overblowing the need for this stuff, whether it's for authorization, or for syncing fragments of the database. I'm sure when you start implementing syncing you'll see that it's just a few simple filters that can be added manually, not much effort compared to overall work to code the syncing solution.
About syncing. Sync framework relies on MS SQL change tracking functionality. As I guess, you're planning central database on server, and 'small' databases on mobile devices (SQLite on Android?), and regular syncing, of a segment of data from server (data related only to user or his 'company'), and syncing back user's work and changes - from pad to server. On the server, you have MS SQL with syncing support and sync framework. Now what about client? how you gonna track changes there, in SQLite? you'll have to manually code everything I guess. As in many cases, having half solution does not help much, the amount of effort might as big as doing everything from scratch.
VITA, on the other way, provides built-in change tracking solution, working for any server. In VitaBooks sample database, find a table log.UserTransaction - it is all transactions (update actions) performed by user(s). The table has 'changes' column which contains all rows (tablename+PK value) for all updated records. One of the purposes of this table and changes - tracking changes for syncing databases, like in your case. It's not difficult to build syncing service using this stuff.
Aug 21, 2014 at 3:12 PM
About 'global filters' - you can do the same thing with EF by overriding the DbSet<TEntity> properties. However, this stops working as soon as you have a query that uses eager loading (like: context.Customers.Include("Orders"))
In that case you'll get all orders even if you specify an additional "where clause" on the DbSet<Order>.
So what do you provide for filtering the "Include pendants" of VITA?

About 'eager loading' - I looked through your test and could not find how this can be done, although I bet this is possible with VITA

Unfortunately, the sync logic is not as simple as you might think. Someone even ported sync framework to Xamarin, which is what we use, after I suggested this. His solution "syncwinrt" (search for it on github) works with SQLite and uses data triggers for change tracking just as sync framework does on MSSQL. So no problems here.

Your idea with the VITA change tracking solution (which was not visible to me - you might document it (better) ;-)) sounds interesting. However, as long as it is not possible to sync between MSSQL on the server and a SQLite db on the client (Xamarin -> PCL or ports needed) I cannot use it. But perhaps I've some time on the weekends. 'could be that I'll look into it then. :)
Coordinator
Aug 24, 2014 at 6:04 PM
VITA does not have 'includes', but provides lazy loading of related entities, like Books.Authors. The main problem with this is that I rely heavily on stored procedures (you can go without them and use direct SQLs), but mainstream is to use stored procs. They provide a lot of benefits, one of the major points - batch execution. So Books.Authors essentially invokes stored proc BooksSelectByAuthorId, and it's not clear how to inject extra condition on the fly. But again, I think you're a bit overestimate the importance of this stuff. Once you have authorization rules enforced on the client, the majority of the queries in the business logic do not need any extras for authorization - following the biz logic is enough, queries are formed OK, and no excessive stuff is delivered. You have to either trust me on this, or try building a real-life project with this stuff - you'll see that it's in fact true. Only very few search queries need explicit injection of authorization clauses.
About tracking. Yes, I do need more documentation, my current problem is that for the most part the design is not final. I had to already throw away several dev docs I've written previously. I will definitely improve on this, and make features more visible, once the overall design is finalized and code is stable.
As for tracking changes - don't underestimate the value of universal solution for multiple servers that VITA provides. We already used it on a real-life project (previous tracking changes version), and it worked great. Knowing from experience that for MS solutions things are NEVER as simple as they promise, I expect you'll have a few heavy bumps on the road with Sync, stitching it together. Yes, syncing is not as simple as anybody would think, but I've done it once, client SqlCE databases syncing with server in the cloud.
SQLite and port to PCL - yes, it's all needed, and it's on my list; currently I'm really busy with db model update refactoring - that part needs more controlled solution for production servers, and it seems more urgent, at least for my current job that I'm paid for. But for anybody exploring and evaluating VITA - that would be a perfect excercise :) you know what I'm talking about...
As for EF and participation - that deserves a separate post.
Coordinator
Aug 31, 2014 at 7:13 AM
Warning - this entry and others following it contain overwelming amount of hatred, frustration, swearing and foul language. Stop reading now if you are not comfortable with these things

About EF and contributing to it.
A couple years ago, when MS open-sourced EF, I rushed to download sources to have a look. By this time I was already working on VITA, after being disappointed with many ORMs I've looked, including EF. I did not have any illusions or expectactions about EF and its bleak future - I was already convinced that EF is total blunder, based on what was already out there, without sources.
Why then I rushed to download EF sources? Two things. First, hoping to learn something, a few new tricks. We programmers learn mostly by looking at other people code. Second, initially VITA was using MS Linq2Sql for LINQ translation, thus limited to MS SQL; I also had to jump through many ugly hacks around it to make it work. So I desperately needed a real LINQ engine, was looking at alternatives (closed and open source), and was considering writing my own. So I was particularly interested in EF LINQ engine - not to copy it, I knew it was not nearly good enough to consider, but just to see another way somebody did this. I've looked already at some LINQ implementations (DbLinq, BLToolkit, some other minor solutions). So I downloaded EF sources, opened Visual Studio, and prepared to spend some time learning.

And what a time it was... Did you see the size of the thing?! it's ginormous. HUUUUGE. Zillions and zillions of classes with cool sounding names and not so clear purpose. I've tried and tried again to navigate through, trying to get the idea how it works, and what is this or that. Again and again I've found myself completely lost, in the middle of some file, with 50 or so opened in other tabs, suddenly asking myself - stop, why am I here, and what was I was looking for? The train of thought was a constant train wreck. After spending several hours there, I finally gave up, closed it to never see it again.
So, call me stupid, that's OK, but the fact is - I can't make sense of this code. And can't contribute - even if I wanted to and was excited to help to improve the thing - and I'm not. One might say I was biased, and did not push hard enough. Well, might be, but consider this - I've done similar thing with BLToolkit, huge thing as well, got through it, and even submitted a couple of fixes to the main tree. In fact, I had an almost working implementation of LINQ engine based on BLToolkit working in VITA - but have to give up in the end. The stop was the same - I hit the wall of 'too-much-complexity' with one of the aspects, and could not figure out how the damn thing works. They were using way too many dynamic lambdas, and debugging was nearly impossible. I had to give up, after spending 2 months or so building VITA LINQ engine and switched ot DbLinq.

One might say: well, EF is a powerful thing, has a lot of functionality, so it's expected to be big. The question is how big. Let's look at some numbers - sizes of compiled assemblies:

EntityFramework.dll - 5056K
EntityFramework.SqlServer.dll - 584K

Now for VITA:
Vita.Core - 238K
Vita.Data - 335K
Vita.Data.MsSql - 42K

Overall comparison is quite startling. EF is roughly 10 times bigger. TEN! Now, feature-wise VITA is at least at the same level of functionality; actually quite more powerful in some aspects - caching, db updates, authorization... So can we explain the monstrous 5.5 megs of managed code by feature richness? I don't think so. It appears that somewhat 80% of EF is just pure garbage. By the way, 5 megs of manage code is really a lot. Here are some more numbers:

System.Dll - 1409K
System.Core.Dll - 230K

EF is several times the size of the .NET Framework - enough said.

And size matters! The difficulty to maintain the code base and to add new features grow exponentially with size. (Naturally - having N times more elements increases the number of possible links/dependencies to N*N). Even simple things or bug fixes - it takes forever to push through. It already shows, clearly, in the dead-dog pace of EF feature progress and problem fixing.
Coordinator
Aug 31, 2014 at 7:14 AM
Size matters even more if you open-source your code - like MS did with EF. After spending several hours in the sources, all I can say is repeat somebody's famous "It's not open source, it's an open mess". (IMHO, opening code bases like these is a total embarrassment for MS.) But let's look at size factor from another angle - 'support' issue.
Yes, many open source frameworks have few, sometimes one maintainer, while EF supposedly has hundreds backed up by MS power and resources. But does this backing makes a positive difference? I really don't think so.
When evaluating a framework to use, your main question is: if I hit a critical bug, how long it will take to find a fix? 2 possibilities:
1 - the framework code is small and understandable, if I need I can fix it myself;
2 -the framework is too big for me to mess with it, so I have to rely on support team to respond, fix and push it quickly.

For EF, I'm afraid fixing yourself is out of question. But I'm afraid case 2 does not work either. If history is any indicator, fixing EF takes forever - small issues or bigger things hang there forever. EF is a monster that moves like a dead dog. As for fixing screwups and shortcomings that are not critical bugs - remember the original non-confidence vote of developers against EF v1? It is outright cry 'fix it!' with a looooong list. How long did it take them to fix this? Still going... And enum support - 5 years or so! That's quite a wait time, and tells a lot.
On the opposite side, a small framework like VITA. One maintainer. Will he be there a year from now - nobody knows. But.. if you see the value, and you see that you can navigate in the code, understand it, and probably fix it if necessary. Then assume it has no support, already! You found this code somewhere, in the dark corner of your hard drive, it is your code from 5 or 10 years ago, from some other forgotten project, or you got it from somebody long ago and just rediscovered. Would you reuse it if it works perfectly for you, saves you a lot of time from writing stuff from scratch, and you more-less can undestand how it works and can fix it if necessary? Think about this...

PS. Just to be fair, most of my grumbling about EF equally applies to NHibernate, and anything ported from Java. Huge mess. So EF is not alone. On the other hand, as I said there are frameworks there of comparable capabilities that you can actually understand (BLToolkit, DBLinq that used as prototype for my Linq engine).
Coordinator
Aug 31, 2014 at 7:17 AM
Warning - reader discretion advised!

Finally, going back to the question of contributing to EF. Let me share a piece of folk programming wisdom, back from Russia where I got my first experiences as a programmer. It is called

The First Fundamental Law of Shit Chemistry

It goes like this:

If you take a pound of honey, and mix it with a pound of shit - you won't get 2 pounds of honey. You won't get even two pounds of bad honey, or very very bad honey. What you get is TWO POUNDS OF SHIT.

Two corollaries:
  1. Order does not matter. Adding shit to honey, or vice-versa produces the same result - shit.
  2. Proportions do not matter either, the result is the same.
You probably already know where I'm going with this. EF is a huge pile'a shit. No amount of honey contributions can turn it into something honey-like. It will remain the same - shit. Fundamental Law of Shit Chemistry.
Sep 1, 2014 at 1:27 PM
Man you've got a very amusing way to tell your opinions. It seems like you've had some very bad days with EF and I can really understand that.
Despite all of your warnings, I am currently quite in a hurry to get startet and, even though I can understand your dislike as I've had bad experiences as well, I just have to use the frameworks that work together most seamlessly and therefore I have to fall back to EF for now.

However, I might come back to VITA when I have (let's see if that will eventually be the case again) some spare time and try to add the features I need. I think PCL, SQLite and Breeze-support would be the first. OData is to complex for me ;-)

And... maybe you should have a talk at a typical Microsoft Conference and show off VITA. I'd love to see the faces of all the EF devs, especially when you start to compare... =D
Coordinator
Sep 10, 2014 at 1:56 AM
as for conferences - scheduled for pres in Dec, right on MS campus:
http://www.meetup.com/NET-Developers-Association/events/206065652/?a=co1.1_grp&rv=co1.1
if you're around Redmond this time - stop by!
Coordinator
Sep 15, 2014 at 1:04 AM
interesting post - about experience with EF6 and SQLite and non-MS databases in general:
http://www.codeguru.com/columns/dotnet/the-sorry-state-of-.net-orms.html
Sep 15, 2014 at 7:54 AM
hehe... no good... X(

Did you tell him he should check out vita? ;-)
Coordinator
Sep 15, 2014 at 4:33 PM
I did, but I doubt he'd take it seriously. His list of of pre-conditions includes things like 'must be Nuget available', and SQLite supported, which is not there. Still, quite sobering testimony, about reality of all these 'supported this and that' claims.
Coordinator
Nov 10, 2014 at 6:49 PM
Edited Nov 11, 2014 at 4:45 PM
hi again
one of your complains about VITA was 'no visual designer'; here's the news - EF7 will come without it:
http://www.theregister.co.uk/2014/10/23/entity_framework_goes_codefirst_only_as_microsoft_shutters_yet_another_visual_modelling_tool/

now they are suddenly discovering that it is just another piece of visual garbage