Are Stored Procedures Good or Bad? And when to use them?

There are a hell of a lot of articles / posts on “Stored Procedures are bad” or “Stored Procedures are good” all of which are written from one aspect or point of view. They never really offer much in terms of when or when not to use them for a developer (or someone at Uni or even for an application / system architect re-thinking things).  You can almost always tell the area of IT in which these people specialise in (or have their most experience) just based on their views on this subject….So what is the big problem? Why the discussion…..

Well, people like to have “golden rules” or “answers”. In this case, there isn’t a particular view that is right, nor wrong, rather only the correct view for a particular organisations requirement. Don’t leave the decision in a data architects hands, nor an application developers (both bring different skills and views to the table). So when embarking on a new project, or a database overhaul, you really need to bring together a number of key personell to ensure you use your database – and stored procedures wisely….There is no golden rule of “Stored procedures are evil” or “Stored Procedures are great”….

Understanding the real pro’s and con’s

Before you can really determine when best to use a stored procedure, you have to think about what are the benefits of using them and the negatives. Ofcourse – this is where troubles arise – some people have arguments that are valid – other ones not so valid. So lets break these down….

Pro’s

Well first off, lets think about learning this rather than going based on “what we have learnt in the past”. So, we can all agree that modularisation and de-coupling is a good thing. So de-coupling our data from our applications can only be a good thing. Great. Now how do we extract that data in the most modular fashion and having minimal reliance on anything else. The answer, a Stored Procedure. The stored procedure provides data architects (and other database roles) with the ability to maintain the database as a seperate entity (as much as can be possible). This provides great flexibility along with other benefits.

Now some will argue this could be done in a DAL (Data Access Layer), which is true, however this means that we have in effect “coupled” our database to another external layer. Now who will maintain this DAL? How accessible and easy is it to update especially if a number of LOBs are dependent on it? In addition, a DAL will be written in what language? Languages change far more frequent than our actual data – many companies will have data that is a hell of a lot older (maybe 40-50 years older) than reletively new languages. To illustrate my point, if you used a DAL written in COM+, how easy would it be for an organisation to find someone to write a web page that utilises this COM+ DAL layer in todays market place? Not great. However, can I find someone who can code that web page and have it interact with a stored procedure – oh, yes I can very easily…

Another pro – is that you can “tune” your stored procedure without really having an effect on calling applications. No code needs to be re-compiled, deployment made etc etc. All that is required is a competent data architect or administrator. This ability to “tune” is because the database doesnt have the overhead of being linked to a particular layer (or GUI if bound).

Performance is sometimes mentioned here, especially as stored procedures are compiled in some cases. Now, in recent years dynamic SQL performance has caught up – however, lets look at the bigger picture of performance. I have worked on a number of projects where a DAL layer was used, and yes, it executed the query almost at the same speed as our SP (most occasions the SP was still faster), however, for more and more complex queries and business rules, the DAL layer became slower. Why? Basically because it used data in-efficiently – bringing some data back, passing it to a business layer to implement rules before making another call to the DB. In such a case – this “technically” is the better solution – with business logic sitting outside the database, however, for performance, placing it inside the database provided far greater performance….

Finally, who actually writes the stored procedures? Often it is someone with extensive experience in this area, someone who can write the SP quite quickly and accurately. Ask yourself who will write it in a business application or DAL? More often than not the junior developer , (that could be a little harsh) or people who dont have as much experience of SQL as they could have – or experienced people who just dont have that much extensive knowledge of the data model / its architecture….In all of these cases – this can lead to poor performance queries and ultimately poor application performance….

So the supposed con’s

Many feel that you have a “vendor lock-in” which on the web – everyone hates (it seems on the web most people want the earth from their software for free and for it to be maintained by the most ethical people in the world who do it to help developers etc and not for a penny..hmmmm). In addition, how often does a company actually migrate to a different database, not often. In addition – if you write your stored procedures (sticking as closely as you can to more standard SQL) then migration may not be such a big issue. To be honest, if you migrate data – if you have a DAL or application that tie to a particular database, good luck re-coding and testing your dynamic SQL there…

Algorithm tuning is also sometimes seen as a negative, stating that millions of people (developers) tune SQL and only a handful can do that for stored procedures. This is really misguided. Developers who write SQL (even in their millions) will not bring to the table anyting more in terms of tuning than a handful of good database architects…

Security…Oh well, this is a classic issue. Many think because it runs in compiled code it is more secure. True to an extent. But how many people (developers) has access to this? In addition, database security is not as bad as it once was, with complex user/role security in place you can really lock down a database and it will only be accessed (admin rights etc) by far fewer people – with less knowing the full database schema. This last point, is probably a negative one – especially if your organisation looses a few good DB men…but its key in such cases to ensure you get proper handovers completed (be responsible as an organisation).

A big, and valid negative, is that you have far greater “power” with dynamic SQL and in a proper language. You can perform greater calculations, applie business rules etc etc. SQL is a little, well restrictive. But thats a good thing – see the next section…

Stored procedures can be highly restrictive – especially when you have a very dynamic database schema. By this I mean one which may well update itself or an application updates it due to business requirements. These are more common than you might think – though by no means are they the norm. In such cases, stored procedures and SQL are far too rigid and lacking in functionality. In such cases you really will need your own DAL…

So when to use a stored procedure and when not to…

As you may have guessed, I am more in favour of the Stored Procedure than not. However, there are those occasions when they just arent a great idea…Its identifiy when to use them, how to use them, and when not to use them for your applications that is key to wheather they are good, or bad for you…

So when / how to use them:

• Use them for basic and moderate complex typical functions (insert, delete, update etc)
• Keep your SPs as standard as possible and not over complex
• Dont be scared to have stored procedures that do contain “business logic” (do not get this confused with application logic – they are different)
• Use SPs, triggers etc to enforce data integrity (your data will last longer than your applications and their chosen language)

When not to use them:

• When your DB schema is more “dynamic” and therefore requires a more powerful language with features to understand how to use it (insert, delete, update)
• When your application requires a lot of application logic from the extracted data
• When you need to utilise a greater / more unique security model on the data (though in this case you may well use a mixture of dynamic SQL and stored procedures to get the job done)

With all of this in mind, to answer are Stored Procedures good or bad? They are great – when you use them correctly…..They are poor when your schema is too dynamic or complex and they are evil when you use them incorrectly….

In browser ECM / over the web ECM

I have been asked to talk a little about browser based ECM solutions, or environments and I thought, why not…First off, browser based ECM interfaces haven’t always been a great hit. In the early days of the web, web based applications were rather clunky, requiring lots of moving around pages to get simple tasks completed. I am not going to talk about the short comings of the web for applications as that is well documented, but, for ECM this environment proved that many web based solutions were slow, hard to utilise and, well, very clunky…

Why are ECM functions hard on the web?

Well the basic functions aren’t that hard these days. Since we have all moved along with how to use the web and our expectations of the web, so have web based ECM solutions – they have improved drastically. However, the problem is that ECM encompasses so much, not just document management facilities, rather the complete enterprise worth of content, in all its forms. Add into this the possibility of Social Media based content and of course Business Process Management (or workflow)  and you can see how this gets more and more complex. I haven’t even touched on extensibility yet either….

So why are these things harder on the web, well they are because of the restrictions the web places on applications. The biggest restriction is the web browser itself, and follow this up with security requirements and you can see why the web becomes almost suffocating for very free content based applications…

The benefits of browser based ECM

Simple, almost no installation on the client machine and the ECM platform can be accessed by any machine with an Internet connection. This means administering the system is a lot simpler and can be moved outside your normal server based type implementations. In theory, if architected well, you will also save on user licenses as the web is “stateless”, meaning you should not have to hold a user license when you aren’t actually interacting with your ECM repository.

However, don’t think you cant utilise thin client type implementations and have your UI in the web browser. You can move web based applications out of the browser with technologies such as Silverlight. This means you get the benefits of the web, without all the restrictions (especially if you choose to run in a “trusted” mode).

 

Good solutions…

If you have and ECM platform that is rather old in its underlying technology (I can’t think of that many that aren’t) you will probably find that their web based solutions are a bit of a “hack together”. The main reason behind this is that technology, programming methodologies etc have changed greatly in the past 25 years, along with user expectations. This doesn’t mean these solutions are bad, rather it means beware that they may limit you in some way compared to newer platforms…

So what good solutions are there that run utilising the web? Well I am not going to list any or do anything like that, rather I am going to suggest that when looking at ECM solutions you think / investigate the following points.

1. Technology used to deliver the interface into the web browser
2. Do you have to run your web application in the browser?
3. Out of the box capabilities / configuration
4. Extensibility of the out of the box type interfaces
5. Distributed processing
6. Integration capabilities
7. Administration

 

There are more, but I want to keep this post from becoming some kind of white paper…

What you will find is that when you get down to these questions – you will find there are still limitations for many of the ECM players when implementing over the web.

 Administration

Many web based solutions are just that, web based. However, administration and the real complexities of ECM are still delivered primarily through a traditional application (which may be installed on the server). To be honest, if you are a web based ECM provider, all features including administration should be capable through over the internet…

Distributed Processing Power

Remember the point of a web based application is that many people can connect to it, it’s available to all that need it. However, some solutions place limitations on the number of users connecting via a web server, why? In addition, some are highly restrictive with regards to what components are installed where, again why? What you are looking for is real capability to share processing power for the system. This can be in the form of P2P (a valued contributor to my posts strongly recommends this – Max J. Pucher), or distributed service architecture (my own preference). Both these methods provide vast scalability and performance and these are key when you think about the web and implementing solutions over the web / intranet…

Application Configuration

Many web based solutions provide a single look and feel and don’t allow much application based configuration. Because of this, developers traditionally built their own interfaces based on customer requirements and delivered these, making the interfaces cleaner, more relevant and incorporating such business requirements as field validation (this is always more evident when looking at web based solutions).  However, this isn’t what I am driving at. Ideally, you need the user to be able to configure parts of their user interface. This could be query forms for an example, or where menu options are displayed etc. The point is, once the user has the flexibility to configure parts of the UI, then their productivity will be increased. This is a key point, especially when we talk about my next point, extensibility.

Extensibility

This is a big big thing. Traditional ECM applications (including those not on the web) provide extensibility through their API, allowing developers to deliver applications that integrate with other LOBs, add business rules etc to the customer’s requirements (within a new application for the customer, not the “out of the box” product). This is a minimum when thinking ECM.

However, the real requirement is that the “out of the box” product, allows business rules and applications to be plugged directly into it. This is so important for ECM based solutions, as ECM within your organisation will grow and include more and more areas of what is termed content. In addition, why not allow the customer to add their own modules in there, or VARs for example, extending the way in which the application and ECM is used…

Plug for workFile Vision

Using the web for ECM is a bit of a passion of mine now, and it is one of the key driving points behind our own ECM platform (workFile ECM – http://www.workFileECM.com ). When working with many other ECM players (as a consultant) I did notice short comings and wanted to get my own platform together that was designed for the web, pure and simple…workFile ECM is a baby, and already we are improving how it works over the web… One of the restrictions to the administration of workFile ECM based applications was the web browser itself, with our own modeller / administration application working in a browser – but in a somewhat clunky fashion.

Things have moved on, and our workFile Vision repository and application takes the next step, staying on the web, but moving out of the browser…

By doing this, we still maintain all of the benefits of distributing an application over the web, however, we also have the added flexibility of running outside of the browser and providing features that can only be made available when running in a “trusted fashion”, such as integrating the web application with thick client applications (take Microsoft Office for example).

In addition, workFile Vision is fully extensible, providing an application framework that allows developers to design new modules and have these plugged seamlessly into the interface. This is to allow the ECM platform to grow with the customer’s needs seamlessly and without developers needing to re-write / re-design modules and applications. Taking this further, all modules can be configured by the user, for example allowing them to design the layout of a repository query form…

Though in the late stages of an Alpha release, workFile Vision 2.0 will deliver everything you would expect from an ECM platform, but much more in terms of the web, extensibility and scalability…Exciting times….I will keep you posted….

Business application UI design

Now this is my first post on this topic (I am sure many more will follow), and I want to talk about some fundamentals with user interfaces within business applications. More importantly, I want to distinguish between UI that looks great in a demonstration, and UI that is great to use…Trust me, they are not the same thing…

Traditional business UI and poor design – it’s not good

When presenting your business application, the first thing it is judged on is the way it looks. It’s a simple fact, if it looks awful then people just aren’t going to love your application even if it’s by far and away the best thing out there in terms of functionality. It’s also worth remembering, that a poor UI will slow down users actually using the system, which is never good – it leads to lost time, poor efficiency, poor views on the system and ultimately frustration.

Typically, business applications UI are functional, and nothing more. All the fields the user needs to access are shown (hopefully in logical places) and the screen is often that lovely grey colour. There is nothing “flash” about traditional business application UI, however, functionality is no excuse for not delivering a great end user experience. This is something that is becoming increasingly more important with business applications – and one of the reasons is probably because of the wide spread use of websites that look great…

UI that presents well, but is it great to use

With WPF and “richer” UI environments (we can include Silverlight in here), UI design for business applications can really add value to the user experience. Screens can become more user friendly, intuitive to use and give the user greater feedback and guidance. However, you can take a good thing too far – and this is something that I have started to see quite a bit of…

Because as designers and developers we have the tools to create something that really has “wow” factor, should we? There is a time and a place – and it is key here to remember what the actual use of the system is, how often users will be using the system (or just screens), how experienced users are and how quickly they can complete their tasks. Let’s look at a real basic example:

Let’s look at searching for a customer’s record. Now I have seen some great and out of the box ways in doing this. Some include browsing and dragging cabinets and records around to allow us to navigate our way to the record / search areas of a system. Others use carousels that act as a “wizard”, with each selection bringing a new set of carousel options (identifying customer type, then account type etc before providing a search screen)…These demonstrate great, they will knock the socks off of the directors and you are a winner….You have put together something different, something that’s intuitive, and something that looks great. Well done…

However, lets now look at this in the real world…I have a user who wants to search for a customer quickly; they may even have the customer on the end of the phone. So is dragging objects around to build a search, or navigate to a search a great idea? Or will they find this restrictive, slow and ultimately frustrating?

Just because we have the tools to build “wow” UI and animations – we shouldn’t feel we have to use them and businesses shouldn’t expect them to be included either…

UI that demonstrates well, but is great to use…

This is where we need to get to. UI that is operationally great – it allows users to work quickly and efficiently and the bells and whistles that are available with WPF, Silverlight etc used to aide in the user experience and bring real value to screens.

Let’s look at our customer record search. We could have a UI that contains a shortcut key to a search panel, which could be slid onto our screen from within any other screen / module. The user can enter some quick key information and be presented in a new tab with search results…The user has in a couple of key strokes called up a search, and found the customer. They can now get on with servicing the customer’s request and then back to whatever task they were working on before. Now this doesn’t look half as flash as our earlier UI, however, it looks good and the use of the “bells and whistles” has added to the system functionality wise – as well as to the user experience.

 

Conclusion…

We have to remember how the user works when designing screens – something that can be lost when going through requirements and what could look / demonstrate great. If your user needs to work quickly, or spends a lot of time in certain screens, then you still can’t beat short-cut keys and use of a keyboard. Touch can help but ultimately, navigating through great graphical based interfaces can be slow and frustrating…Flash new developer and design tools are there to help, and should be used when needed, not for the sake of using them…

For business applications the design rules should still be, “what works quickly for the user” and then, “how can we make that look and feel better”…

Redefine the way we use the web, to unlock its potential…Web 3.0?

This is something I have been thinking about for a number of years now, but more so recently with a lot of talk of HTML 5. Basically we haven’t really changed the way we use the internet (from a technical point of view) since the web became mainstream shall we say. Sure, we now use it in new ways which we hadn’t dreamed of (habits and the way we communicate with each other), but essentially the web still works the same way it always has. We use the web as content rendered as HTML that is displayed back to us in a web browser. Even if HTML 5 is the magic version and delivers so much more in terms of animation and streaming has it actually changed the way in which we use / the web works for us? No…

Let’s not go back to the good old Mainframe environment…

It seems more and more IT professionals and large organisations see the web as the new mainframe, especially when you start talking “thin client” and “cloud computing” (the cloud could be seen as our mainframe..scary). When you start looking at mainframe environments and then cloud and thin client computing, you see that the basic concepts are very similar. So what do I mean, well, all of the processing happens on a server, the machine you actually use to access it, doesn’t really have to do anything. In a mainframe environment we have dumb terminals, in the new way of thinking (trying not to laugh, sorry) we have a PC that run’s a browser (this could be a very low spec machine), and if all we did is “cloud compute” we perhaps wouldn’t need anything else?

Sure I see benefits, some of which are green, but the negatives are so obvious to see. These are essentially the same problems we have with mainframes and the same problems that lead us to using the “PC” and the “Network” to replace mainframes?

Some thin client issues?

Let me give you an example. Imagine you and I are working as designers, creating 3D computer models of pretty much anything. We may even be responsible for animating these 3D models (think something like toy story, I don’t know why, it just popped in my head). Ok, now imagine you are part of a team of say 20 working on these models, of course you are designing Buzz, someone else Woody etc. Let’s think just how much “processing” power do we need for this – just you and your requirements? The answer, quite a bit, well a lot. Now image having to times that by 20. Oh, and now let’s have that processing carried out in a “thin cloud computing environment” (of course your application is written with the fab HTML 5 which means we can do anything), which at the end of the day needs a hell of a lot of work going on at the server, oh and traffic across our network… Do you see the problems?

Well basically, even with the advances of our hardware, the server will be doing too much and things won’t go well. The system will be going slow, maybe crashing, you as a designer will be going mad with frustration, along with the rest of your team, oh and not to mention you are working to a deadline so the project manager is now going mad. Let’s throw into the mix too, that our team is distributed across the States and the UK, and some of us love using Internet Explorer, some FireFox, some even Chrome…Hmm though in theory the web is great here, it is no match to a good old client desktop, some distributed servers…

Now I know I am focusing here on a situation that doesn’t lend itself to “cloud computing” or “thin clients” but if we believe all the hype of HTML 5, cloud computing why shouldn’t we be thinking this is possible? But, as our hardware advances so does our software (though at a slower rate granted) and we as users (be us general public users or business) expect more and more performance and capabilities. So while some of our user requirements do seem to lean us toward a cloud computing way of working, soon our requirements will no doubt swing back the other way (and wont we be repeating the Mainframe and PC story all over again?)

There is an answer

The answer is pretty simple to be honest and it is something Flash showed us the way to a number of years ago when it first started popping up on the web. The answer is a mixture of the two.

So let’s start evolving how we use the web properly (not just our habits) but how it is used. The web becomes a communications network and in some ways returns to its roots. We can still use it in the way we are used to, as in we find websites and we view them in a web browser, however, those websites that aren’t just presenting us with some information, or basic shopping facilities, websites that are more “applications”, get themselves installed on the client machine. So think MS Office on the web. Why install on the client? So that the user experience is not restricted by the web architecture, nor the browser, and that “processing loads” are removed from the server and distributed back down to the client PC.

Isn’t that what Flash was doing, installed and running on the client, err years ago? Yes, and that’s why Flash has worked so well to now…The problems with Flash are not what it visually looks like, nor its basic architecture (running on the client), the problems are that it doesn’t lend itself to being able to deliver “applications”. So it is great for the web to show animations, and funky banners, slick movies etc but don’t think it will be great at delivering that 3D modelling tool we spoke about earlier…

So let’s go back to our 3D modelling requirement in the designer’s studio. In our new web world we are now working with a RIA that actually runs on the client machine, uses local storage on the machine and uses the web only for bare communications and maybe storage of files that are to be shared. All of a sudden, all of the issues with “thin client” and “cloud computing” and server loads are removed, yet essentially we are still using the web and “cloud computing” to an extent…

So the answer is RIAs that use the client processing power and that do not run in the web browser.

Is this available…

Yes it is. Since Microsoft launched its Silverlight platform (which many see only as a competitor to Flash) it has been working towards this type of scenario, where we can maximise the benefits of the PC and the benefits of the web and cloud computing. Silverlight 3 was the first version to deliver an out of the browser experience and this has been taken further with Silverlight 4, with it being able to run as a trusted application on the client machine. Oh it also runs on Mac’s and PCs and if in the browser, any browser…

Silverlight, though in some ways similar to Flash and even the old Java Applets, is a new way of using the internet, rather than us re-inventing the same way of using the web with more bells and whistles. Like flash and Java applets, Silverlight essentially runs on the client PC. Which means we can utilise its processing power to do our work, it doesn’t need to go back to the server for updates to the UI, business rules or anything like that, and it can be done there on the client machine? However, it is connected and delivered essentially through the web as a communications network, so its data and files can be easily pulled and pushed across the web and stored there. Updates to the software are also delivered through the web, with the user being able to get the latest versions of the software just by using the software itself.

At present this is all still young, but the potential is there to change our web experiences and what we realistically should be using the web for. MS Office could be delivered as nothing but a Silverlight OOB (out of browser) application, allowing us to purchase it online and using it within moments. And it would look and feel just like the version we currently have from a CD (not the slightly less functional web version). Business applications could be delivered through organisations intranets, or their “cloud providers”. Websites that provide “secure” trade or partner areas would essentially have these installed on the client machine. Twitter, Facebook and other types of highly interactive websites would be delivered as RIAs installed on the machine (there is a prototype for Facebook already built and made, which you can download and use at http://www.silverlight.net/content/samples/apps/facebookclient/sfcquickinstall.aspx). You havent used the flexibility of the web at all, if you were on a new machine and wanted to get to facebook, still visit the website where you would get prompted to install the client, which would be a simply and quick install…and away you go, back on facebook.

The future then is…

Re-defining the web as a communications network and moving RIAs out of the web browser and down onto the client. By using the web in this fashion we get a truly distributed environment that has the benefits of the web, but also the benefits of the client machine…

Can Silverlight save Windows Mobile?

I love the Silverlight technology and since I first started working with it (back in Alpha 1.1) I have always wanted to see it working on my mobile phone. Could Silverlight 4 be the release that finally makes it to our mobiles?

iPhone?

In a recent post I looked at Silverlight being used on the iPhone, allowing streaming of video content over the web to the iPhone. (http://andrewonedegree.wordpress.com/2009/11/27/silverlight-on-your-iphone-even/ ) I took this as a big positive move from Microsoft that they are seriously looking a Silverlight being used on the mobile phone. The iPhone is currently seen as the “bench” mark for smart phones (though a lot of other manufacturers offer far more features and functions in their phones) so for Silverlight to be a true mobile success, it will need to be able to work on the iPhone as well as Windows Mobile. We should also include other popular mobile platforms, such as Android etc.

Windows Mobile 7

For some time now, windows phones have been stuck using v6.5 which is based on the Windows CE 5.2 kernel, which was the same that powered Windows Mobile 5 (so that’s back to 2004). It has meant that while Windows Mobile phones still offer great office integration etc, they lack that certain wow factor, which is seen in abundance on the iPhone for example.

One of the reasons why the iPhone is so popular is the vast number of applications yo

Silverlight delivers RIA and on your Mobile?

u can get for the phone, which is brought about simply by the vast number of developers out there for iPhone applications. So for Windows Mobile 7 to really compete, it too needs access the development community and have vast numbers of applications being developed for the platform…This is where Silverlight can step in…

Silverlight 4, the platform for building Windows Mobile 7 apps?

Looking through a recent ChannelWeb blog, it seems that a number of people have stepped forward and indicated that Windows Mobile 7 applications will be written and delivered via Silverlight 4. This makes great sense as it opens up the mobile platform to vast numbers of .NET and Silverlight developers. It also provides a real useable platform for business applications to work on multiple devices. (Something I have been looking into myself – http://andrewonedegree.wordpress.com/2010/01/07/ecm-access-on-my-phone/ ) My own company uses Silverlight to deliver a number of solutions, so having the capabilities to re-use our Silverlight and .NET code, to put-together mobile versions of these powerful business applications would be highly desirable…

It’s also worth pointing out that Silverlight development is moving along rapidly, with v3.0 being released back in July 2009 only for v 4.0beta to be launched just 4 months later in November. With this rapid development of the Silverlight platform, we are also starting to see more and more businesses and software providers switching and using Silverlight as their technology of choice.

The benefits go on, by using a “web” technology such as Silverlight, you don’t need to limit your applications to a particular native SDK, which is what you have to do for the iPhone or Android.

Conclusion

Is Silverlight the silver bullet that will make Windows Mobile 7 competitive and cool again? Well it will go a hell of a long way to helping, however the underlying operating system will have to run smoothly, be highly intuitive, support touch without the need of a stylus and above all, look great while still providing all that office integration we expect from a Windows platform. If Windows Mobile 7 can deliver this, then Silverlight will ensure a raft of developers get involved building cool applications, not just for the general public, but for businesses too…

Watch this space.

Intelligent BPM maps

I have posted about BPM maps hindering flexibility and capabilities to some extent before (regarding systems integration). See:

http://andrewonedegree.wordpress.com/2009/11/30/bpm-mapping-tools-integrating-data/

http://andrewonedegree.wordpress.com/2010/01/08/incorporating-automation-into-your-processes/

However, in this post I want to take this further by looking at how BPM maps (I use the term map loosely here) can become intelligent and hold much more than just business process routing rules…

The role of the map

For many this is the “definition” if you like of a business process, shown in a graphical map format. This is great, and it’s true to some extent. However, I believe the primary role of a workflow system is to deliver systems integration, not a predefined diagram of a process.  BPM and workflow only works well when it brings together systems, people and data to maximise the efficiency of a business requirement (or process if you like).

So what is the role of the map? Well it is there to provide business rules for a cross section of applications to deliver a solution that allows users to do their work effectively. (Not easy to read that sentence). This work is shown as a process. For me, I prefer to see processes graphically, but not in my BPM system, well not used to define the rules etc within my BPM solution. Graphical representations are great for identifying the business requirement, and should be done by a BA. But process maps should be used as a “specification” if you like for a developer to build my intelligent process map…

Using a developer to implement my business rules

I know that many of us want to have a nice mapping tool that allows a business analyst (BA) to create and modify maps / business processes. However, in the real world, this means you have a couple of restrictions / issues.

1. You can’t easily integrate with other LOBs and data required for a particular step
2. You can be limited to other business rules / factors (that are outside the scope of your map)
3. Automated steps often require “Robot” type step applications to be written (specifically for your requirement)
4. Much more emphasis is placed on developers for the actual implementation / front end of much of the system (if you require intelligent integration / more complicated system integration)

As mapping tools get more powerful you still have these issues, mainly because a BA is just that, not a technical person who wants or should be bogged down in the technicalities / functions / calculations etc required for the business.

By using a developer to take your map and build business rules into a BPM system (if your BPM architecture allows this type of process definition), you open up a world of systems integration and flexibility. Effectively your business rules / map can now become intelligent.

Intelligent maps

An intelligent map is more than just business processing rules. It contains actual business processing logic, it has the capabilities to bring in data from third party software, carry out complex calculations and functions, raises events and triggers and does all of this within the map itself.

 Most BPM maps cannot provide this level of integration or capabilities to execute / carryout processing functions. Many times these types of functions are provided in the form of “Robot” applications or step processors. These are background applications or services written by developers to include business rules and functionality into the process map, because the map itself cannot support this level of intelligence. The outcome is a solution that requires much more processing power, requires greater input from developers and one that is harder and more costly to maintain.

By shifting emphasis of functions and rules to an intelligent map, you provide a BPM solution that delivers greater out of the box functionality, keeps initial costs far lower and requires less development work / bespoke step processors to be written. In addition, when your business needs to adapt and change, updating processes are far easier and quicker. Since the map itself contains the business rules of your processes (as well as the definition of that process), you need only modify one thing, your intelligent map. There are no background processors that need modification, no new application changes to be made etc. Because the business intelligence is all stored in a singular place…

Quick example…

A good example of a BPM platform that works in this way is workFile BPM. It has been architected to ensure the “map” holds all the business rules as well as having the capabilities to integrate with other LOBs and execute functions, triggers etc within the map. Developers have to build the map in this case, based on information provided by BA’s.  

The out of the box user interface is in most cases the only interface you need, simply because of the intelligence available at the map level. However, there will always be occasions when “bespoke” processors are required, and the workFile BPM platform provides a complete XML Web Service API in which developers can build on the intelligence provided in their maps within workFile BPM…

Conclusion: System integrator or process definer…

I see the main aim of BPM and workflow to raise the efficiency of businesses by making it easier for users, and the business, to complete work. Defining processes allows us to visualise this work, however, the BPM platform brings together everything that is required to complete the work. So a BPM platform should be a systems integrator first and foremost, this is the real beauty of BPM and workflow…

Incorporating automation into your processes

This may seem quite simple, however, it is often something that is neglected somewhat. When designing your processes, ask yourself just what can be automated, and how much automation will add to the efficiency of that business process.

Identification

It can be tricky to identify everything that can be automated straight away. Only after a good analysis phase will the majority of processes / tasks that can be automated, be identified. Some of these will be obvious candidates for automation; typically these are calculations or actions that can be completed with all the information stored within the BPM system. Other automation candidates may not be so obvious. The less obvious processes / tasks are often overlooked because of the way the process is currently worked, typically requiring integration between systems (maybe even multiple LOB applications). It is always important to try and automate as much as possible, or at least indentify everything that in an ideal world, could be automated…

Restrictions

Once you have identified all your possible automated “steps”, you really need to see what you can realistically automate given your current BPM technology, LOB application integrations capabilities and of course, your budget…

One of the big problems with BPM modelling tools is that they can become very restrictive in what can be achieved with regards to integration. This is something I have blogged about in the past, http://andrewonedegree.wordpress.com/2009/11/30/bpm-mapping-tools-integrating-data/ with many automated “steps” you will require the services of a developer, hopefully your chosen BPM platform will support this kind of integration and processing…

The next hurdle is to identify what integration capabilities your other LOB applications provide. If you cannot integrate with them at all, then your step cannot be automated and will have to rely on some good old fashioned user processing power, not so efficient. Having a good IT department or use of a good IT consultancy typically means that your company will have a good and clear understanding of its IT and have some form of strategy / roadmap in place. If so, you will probably find that your LOB applications (unless very old) will provide some form of API allowing integration possibilities. (Ideally your business will have proffered technology platforms, such as .NET, Windows etc). If this is the case, then you can start to investigate just how much integration is possible and evaluate the costs involved in automating your process step…

The benefits

Automated process steps provide a number of benefits, the main two of which are:

1. Efficiency
2. Accuracy

There are obvious efficiency gains by automating a step, which obviously raises the efficiency of your process and improves SLAs etc. However, accuracy is often overlooked. Automated steps are far more accurate (once they have been fully tested) as they simply remove human error from that particular step. Now I am not saying this means your process will not have “issues”, but what I am saying is that an automated step removes user error from that particular part of your process, something that can be very time consuming.

With these two main benefits you also get a great “cost” benefit. If you measure your time and resources and place a monetary value on these, you will soon see a clear ROI timeline for automating a particular process step. This will typically be the deciding factor (if possible) in choosing to automate a particular step or not…

Conclusion

Whenever you review your business processes, even if you don’t have a BPM system in place, always ask yourself which processes or “steps / tasks” could be automated. Don’t feel restricted because a process spans multiple systems, departments or geographical spaces just indentify candidates for automation. A good way of doing this is by using a good independent consultant.

Automation is a great way of raising efficiency, accuracy, productivity and reducing operational costs. It there is always in the benefits of a company to automate as much as possible…

RIA is for your Intranet

Silverlight delivers RIA

Many corporate intranets require typical web functionality, but also they are often used to deliver application type solutions to the user without the hassle of maintaining any installations on the client PC. So what’s my point….Well my point is that web applications provide great value to administrators of systems, but they really do detract from a user’s experience. So how do we make what should / could be a thick client application, work well within an intranet environment?

User Experience

We all have used web based applications, some great some very poor. However, with almost all of them they are delivered in an old fashioned way. For example, a user makes a request to the page, something happens on the server, the page is recreated at the server then sent back to the client updated in its new state….Great. However this can take time, which isn’t always great, but more importantly it can become limiting and frustrating to users. You also have to think that changes to the look of a page still have to be processed and completed on the server. At the end of the day, standard web applications still feel, well, a little “clunky”, no matter how much fancy Javascript you can write…

Often because of this, users don’t give great feedback on the intranet itself, which is wrong; rather they should be giving negative feedback on that particular application / service delivered through the intranet (there is quite a big difference).

An improvement…

Intranet applications that take advantage of AJAX do deliver a far greater experience – AJAX has ensured we are moving in the right direction, ensuring you don’t always have to wait for a page to be submitted and then returned – rather the page has a “conversation” with the server. This has improved the user experience greatly and has removed much of the “clunky” feel web applications can have. But there are still issues. Here are a couple of things I don’t like to be honest:

1. It still looks and feels like a web page (which can be a positive – but for more complex applications this is a big negative)
2. You are still navigating around and waiting for a web page to come back (when requesting new pages)
3. Users can request “back” in the browser (unless the application has turned this off) or worse they think they should be able to go back…
4. Limitations on the application behaviour
5. Occasionally the server is still being used to deliver client side features (conversation that attracts data then updates a data grid on the web page)
6. Flexibility as a developer / ease in which to control the application

So let’s be honest, a web application still cannot compete with a good old fashioned “thick client” application installed on the PC for usability / user experience and performance….

The solution?

Well RIAs (Rich Internet Applications) hold the key. They bridge the best of both worlds in regards to thin client administration and thick client usability. This is more apparent for an intranet solution; there really is no excuse for keeping to old html based solutions.

In an intranet environment you can really go to town. You can deliver massive sized RIAs down to the client machine as you are on the network. More importantly with a little “caching” by your browser, the application need only be downloaded when an update has been made to the software.

Silverlight provides a great RIA platform, and allows intranet solutions to deliver the best of both worlds to businesses. Silverlight has many features built in that allows developers to manage how the application is delivered down onto the client machine, parts of the application can be split up and downloaded in the background ensuring the user doesn’t have to wait for areas of the application. Basically as a developer, you have almost a “thick” client environment to play with, allowing applications to behave just like thick client applications and provide the user with a far greater experience. This ensures processing that is done at the client end (such as displaying data) still all occurs on the client machine – with the client processor doing the work). There is a real separation between client processing, data, and server interaction (especially as the application utilises a Service Orientated Architecture)

However, you can also utilise the benefits of typical web pages, allowing state to be stored by the browser, navigating around in pages (if desired) and more importantly, not having to worry about client installations.

An issue – there can’t be?

There is just one problem with intranets being written in Silverlight or demanding Silverlight based RIAs. Most in-house IT departments will find they have any number of people who can “play around with HTML” or claim to be a “web developer” when really their skills are limited to modifying standard HTML. Even with typical ASP.NET and AJAX solutions, you will find that people with a basic understanding of HTML will be able to modify certain aspects of a web page / web application. With Silverlight, you need skilled individuals. This means companies have three options

1. Employ people with the skills (not cost effective)
2. Provide training for current employees to get some basic skills (again is this the best option?)
3. Utilise software companies on an ad-hoc basis (best on a retainer of some sort).

 

The Silverlight RIA payoff

Silverlight provides the possibility for intranet solutions to work as well (if not better) than typical thick client applications. This ensures usability, flexibility and scalability of solutions and opens up a new way of interacting with business applications through your intranet. There is no reason for intranet solutions to deliver “clunky” applications that users will not like to use, instead, provide “real” applications that empower your staff to do their work while keeping your IT administration to a minimum…

Bing Maps getting Silverlight

Well it had to be just a matter of time before Bing Maps started using Silverlight to deliver the richest mapping experience on the web. Since the start of November I have been playing around with the Silverlight Bing Maps control which far out-performs the AJAX control and for me the HTML version on the actual Bing website.

The payoff

Microsoft's Bing

Well for the end user, the Silverlight experience is far smoother and allows you greater control. For example zooming into an area on the map using the wheel of your mouse is a nice touch, but the app renders smoothly. In addition you can mesh together traditional map views with aerial photos. There are also nice features such as the street side walker – which currently isn’t available in that many areas, but it allowed me to jump into the map and then walk around parts of the world I have never seen (well visited now….).

With the Silverlight version, everything just feels so much more professional, it’s a real jump forward in terms of the functionality that is capable and the experience it provides the end user. Why not see for yourself, you will need to have Silverlight installed. http://www.bing.com/maps/explore/ NB If you don’t have Silverlight installed I suggest you get it asap.

Microsoft has also followed on a trend of offering “app stores”. A new application gallery will be available allowing developers to include their own information on a map.

The biggest pay off though is the capabilities this provides to other developers and websites that want to use mapping technology. I have already seen a number of demonstrations showing how you can overlay / highlight “areas” within a map. One great demonstration shows the New York marathon route, it not only shows you this route and the “area” covered on the Bing map, but also shows runners moving along the route – comparing their relative times etc….Not bad….

The .NET framework…

I have to say that I like the way Microsoft is going, building everything on the .NET framework or a subset of it. It allows more powerful applications to be built and integrated with each other. This is another great example – Silverlight, which is a subset of .NET with a WPF subset as a presentation layer, combined with the Microsoft Live web services (again delivered in .NET) delivering a feature rich experience for users. More importantly though, working in this way provides the development community with the tools they require to take things further.

By combining the Silverlight Bing Map control with the Microsoft Live web services, it is now a quick and rather simple(ish) task for any .NET developer to deliver powerful mapping / mapped based services to clients that look and perform great.

Integration is Key (ECM / BPM / Social media)

For many years I have waved the banner for single application experiences for end users. If you can deliver a single application that allows the end user to carry out all their work, gain access to all the files they require, interact with many other LOB applications (without knowing it), just think what a positive impact that would have on any organisation. Think how better informed that user will be, how much improved their decision making will be, how much customer services will be improved along with customer satisfaction, and also, think how much of a gain that organisation will make in efficiency, productivity and ultimately profitability…

Integration has long been the key to this ideal, and ECM and BPM often show how this can work, integrating with key LOB applications.

Problems…

The problem is that people want everything to integrate without putting any effort in. This means that organisations spend a lot of money in getting applications to integrate with other companies applications and software. While this can be great for the customer (if you have the same selection of applications and software) it isn’t always practicle. Throw into the mix different operating systems, different versions of software and the daddy of all, different business requirements from that integration….All of a sudden you see how muddy the water can get and just how complicated system integrations can be, and why that single application experience is so hard to achieve…

Progress

With the bright invention of XML has come a whole host of ways of integrating applications. It has provided the bridge between old COM and COBRA components, interopability between application components, and most importantly, delivered us XML Web Services and Service Orientated Architectures (SOA).

I love XML Web Services and the capabilities these alone can open up to organisations. If applications deliver good APIs through web services, then integration is made so much easier, be it integration “out of the box” with connectors, or more efficiently through actual developers and professional services.

Is Social Media leading the way here?

Yes…There you go, a nice short answer. Basically Social Media is leveraging web services (especially RESTful services) to allow integration between web sites / applications. Take the recent joining of forces of LinkedIn with Twitter. LinkedIn can now pull in your “tweets” and have these shown as status updates within your LinkedIn profile. Now think back to a business environment and you can see how using one application therefore effects data / content on another application / area of the business. This type of seamless integration is what adds real efficiency gains across an enterprise.

One Degree of Separation

When I founded One Degree Consulting, one of my main aims for the consultancy was to be able to provide consultancy services and solutions that delivered a single degree of separation between the end user, the data / content, and the functions they required to do their job. This may sound a little idealistic, but it can be achieved and should be the goal of business decision makers in all organisations. To be blunt, to achieve this, application integration is key and should be at the forefront of any decision making when it comes to IT based projects and solutions.

If Social Media sites hadn’t have seen how powerful joining forces could be and had maintained a closed API that couldn’t easily be integrated, then the whole point of Social media and sharing may well have been lost….Businesses, take a leaf out of their book, think integration for everything…Its key….