Allowing Business Users To Program Your System Is A Recipe For Disaster

0 comments suggest edit

I don’t know about you, but every company I’ve ever worked at had a Fort Knox like system in place for deploying code to the production server. Typically, deployment looks something like this (some with more steps, some with less):

  1. Grab the labeled (tagged) code from the version control system.
  2. Obviously, ensure that the application must compile.
  3. Another developer other than the author must review the code on some level and sign off on it.
  4. Automated unit tests must pass.
  5. If they exist, the automated system and integration tests must pass.
  6. The QA team tests the application and approves it.
  7. The deployment engineer (typically a developer or QA person) very carefully deploys the application attempting to avoid any downtime.

Interestingly enough, many of these companies didn’t have the same procedures for other documents and systems used to run the business. For example, one could in theory login to their CMS system and change the home page of the site to contain every expletive in the book just for fun and it would show up immediately.

Spreasheet photo by
http://www.flickr.com/photos/caterina/ There are a lot of people who want to make it so that the business user can write code by connecting legos. The typical examples include dynamic rules engines and their ilk. Yeah, let’s let Joe the finance guy tweak the rules on the rules engine on the fly by drawing lines and connecting boxes.

The problem with approaches like this is that it ignores the fact that the effect of these changes is no different than writing code, but often with much fewer checks on quality before it gets deployed to where it can do damage.

These systems often are lacking:

  • Version Control
  • Backup and Restore procedures
  • Quality Assurance testing
  • Formal Deployment procedures

A recent report (via Reddit) illustrates this point with a list of news stories on how errors in spreadsheets have cost businesses millions of dollars. A couple of telling snippets (emphasis mine). This one on the lack of version control and auditing:

http://www.namibian.com.na/2005/October/national/05E0F49179.html\ The Agricultural Bank of Namibia (Agribank) is teetering on the edge of bankruptcy. “There is no system of control on which the auditors can rely nor were there satisfactory auditing procedures that could be performed to obtain reasonable assurance that the provision for doubtful debts is adequate and valid,” note the auditors. Auditors found that its loan amount to the now defunct !Uri !Khubis abattoir changed from N$59,5 million on one spreadsheet to N$50,4 million on another, while the total arrears was decreased from a whopping N$9,8 million to only N$710 000.

And this one on the lack of training and Quality Assurance.

Only a matter of time before the spreadsheets hit the fan

  • Telegraph (UK), 30 June 2005\ In his paper “The importance and criticality of spreadsheets in the City of London” presented to Eusprig 2005, Grenville Croll of Frontline Systems (UK) Ltd. reported on a survey of 23 professionals in the £13Bn financial services sector. The interviewees said that spreadsheets were pervasive, and many were key and critical. There is almost no spreadsheet software quality assurance and people who create or modify spreadsheets are almost entirely self-taught. Two each disclosed a recent instance where material spreadsheet error had led to adverse effects involving many tens of millions of pounds.

The solution is not to make programming more like the way business users work now. The solution is to apply the lessons learned from software development into other business processes.

In the same way that companies rely on heavily trained developers and rigid deployment procedures in place for code, companies should make sure their business people are just as heavily trained in the software they use on a day to day basis. After all, million dollar decisions are based on the content of these systems daily.

For example, spreadsheets should be version controlled. Changes to rules within a rules engine should have to pass some automated tests and manual QA before being deployed. All of these should be peer reviewed.

Found a typo or error? Suggest an edit! If accepted, your contribution is listed automatically here.

Comments

avatar

16 responses

  1. Avatar for Jon Limjap
    Jon Limjap June 3rd, 2007

    Tsk, this happens a lot to companies who call their Excel spreadsheets "databases", super-denormalized ones at that. My wife used to work for a shipping line and one aspect of her job was that when information regarding customers or consignees changes, they have to search through the whole spread sheet to apply the changes to each instance.
    Talk about the potential for errors.

  2. Avatar for The Other Steve
    The Other Steve June 3rd, 2007
  3. Avatar for Brandon
    Brandon June 3rd, 2007

    Phil,
    I take it that you have never taken over applications that were developed and built by standard users? I have the experience of dealing with both the Access applications and also VB applications that were written by a user who took a class. The user is trying to do something to improve their work flow, but in turns makes it a living nightmare to maintain and support.
    Jon mentions users using super-denormalized Excel spreadsheets. I have had to deal with Access databases that reach up to 1GB in Office 2000 and 2GB in Office 2003 of super-denormalized data. I guess there are just some cases, which people just want something that works to get the job done rather then something that improves their job.

  4. Avatar for hgs
    hgs June 3rd, 2007

    <p>Well, I agree that version control is a good thing, and testing of course. But 7 steps which are manual steps, even if each such step invokes an automatic process, is, frankly, ridiculous.</p> <p>I tried to get some researchers and academics to use version control in developing a website, collaboratively. Given that most of them were used to Frontpage, presenting them with even the GUI based tools for CVS or Subversion was too much. I never found a system that was simple enogh to use, fitted in with the Windows Metaphor sufficiently, and was easy to explain. Now we have more flavours of version control (CVS, svn, darcs, git, rcs, ...) than you can shake a stick at, and I can't think of any that I'd consider presenting to a non-programmer.</p>
    <p>These things are important, so they should be easy to do. And I don't mean easy for developers, I mean easy for people whose fear level is at: "If I make a mistake, the computer will blow up like it does in the movies!" And if you think I'm overstating the simplicity level, then why aren't more people using troff or LaTeX to produce documents?</p> <p>I think we need User Interface and design experts to wrap a decent Facade Pattern around these tools.</p>

  5. Avatar for hgs
    hgs June 3rd, 2007

    I thought 'p' was an allowed tag. What did I miss?

  6. Avatar for Haacked
    Haacked June 3rd, 2007

    <p>@hgs: so did I. I better double check this.</p>

  7. Avatar for Wesley Shephard
    Wesley Shephard June 4th, 2007

    We use a lot of business driven DSLs. However, we never even *considered* making changes by the business unit "live" wihout the traditional QA process being applied.
    There is something to be said about the versioning though. Each data driven DSL process has had to "reinvent" tagging and releasing versions. Usually via a "copy existing version" and then modifying and "releasing" the modified version... which doesn't support concurrent changes except insomuch as multiple unreleased copies can exist and we can difference data.

  8. Avatar for lb
    lb June 4th, 2007

    All good points. and yet...
    The definition of "Programming your system" is ever-changing.
    There's no clear distinction between Data and Code. So while users can enter data and this data is used to determine how code runs -- they are in effect programming the system.
    Hence I think a more concise title for the base problem would be:
    "Allowing Users Is A Recipe For Disaster"
    I think the kind of in-built revision control mechanism that they have in wikipedia (for example) is the way forward in terms of bringing "version-control" to ordinary people.
    And the difficulty is that implementing a system for lower-skilled users requires higher-skilled developers.
    The emphasis is not on taking abilities away from users, but on implementing them in a way that allows for course corrections, dead easy revision control (and merging), easy auditing, usability, -- all those sort of things that look easy but aren't.
    It's inevitable that these things are rare: because the end user doesn't realise the value of them until it's too late.
    good food for thought today. another helping please.
    lb

  9. Avatar for Scott
    Scott June 4th, 2007

    "I have had to deal with Access databases that reach up to 1GB in Office 2000 and 2GB in Office 2003 of super-denormalized data."
    I can walk over to one of our analysis labs (I work for a cancer research center) and find a stack of 100GB firewire drives stacked 10 high filled with Excel spreadsheets and CSV files.
    At the heart of EVERY organization there is either an Excel spreadsheet or a rinky-dink Access database that the entire company is in some way dependent on. Indie consultants call that the "cash cow".
    I'm currently trying to talk our team OUT of the lego brick design phase now. I copied over ALL of our production databases and unzipped they were a whole 2GB in size.

  10. Avatar for Haacked
    Haacked June 4th, 2007

    @hgs - It turns out I must've changed my configuration when I last deployed it. I fixed it just now, so the p tags work again.

  11. Avatar for SimonTeW
    SimonTeW June 4th, 2007

    Interesting that the commenters above never seem to have heard of Document Management Systems (DMS) such as DocsOpen. They handle version control for documents at least as well as any software version control system.
    I used to work for one of the large law firms in the City (of London). Every single document (Word, Excel, Visio, whatever) was automatically added to DocsOpen when it was created. It was not even possible to create or access a document outside the DMS since users did not have rights to view their own C: drives and the Office applications were heavily customized to lock them down.
    Not only did the DMS handle versioning, it also recorded every single time a document was accessed (not changed but accessed!), and by whom. One IT guy I worked with got fired after the access logs showed he'd used his super-user privileges to look at restricted documents related to staff salaries.
    Access rights could be restricted in exactly the same way they can be with a software version control system (to individuals and/or to groups). Rolling back a document was a breeze and it was even possible to fork a document or merge a fork back into the main branch (admittedly this functionality was limited). The DMS also handled the situation where users were working off line and could seamlessly merge a modified local copy of a document back into the document repository.
    Show Differences (between different versions of a document) was far more precise than anything I've seen in IT. Law firms need to be able to track changes to a document down to the character level (moving a comma can change the meaning of a legal document). I still miss DeltaView (the application used by most major law firms world-wide to show differences) every time our crappy IT version control system tells me a whole line of code has changed when I add a single space or something.
    Admittedly all this relates to law firms, not financial institutions. From friends who worked in financial institutions in the City I know they have a totally different culture from law firms. However, it's certainly not right that all non-IT companies are sloppy regarding version control.

  12. Avatar for Stefan Tilkov's Random Stuff
    Stefan Tilkov's Random Stuff June 4th, 2007

    Phil Haack: I don&amp;#8217;t know about you, but every company I&amp;#8217;ve ever worked at had a Fort Knox like system in place for deploying code to the production server. [&amp;#8230;] The solution is not to make programming more like the way business users work now. The solution is to apply the lessons learned from software development into other business processes....

  13. Avatar for Karthik
    Karthik June 9th, 2007

    This concept is one of the main driving motivations for people selling Business Rules Engines. However even these BREs don't do much to mitigate the inherent risk involved in allowing business users to control software functionality without the traditional QA process.

  14. Avatar for Rajgo
    Rajgo October 4th, 2007

    Hi Phil,
    I enjoyed your post.
    Letting the business types mess directly with the system is dangerous. 100 points to you.
    business guys know what the software is supposed to do
    software guys know how it is doing it
    There are 2 issues as I see it.
    1. When the shit hits the Fan, how do the IT guys and the business types collaborate easily to resolve the issue.
    2. How do you minimize the impact of testing changes to your business rules on the rest of the system.
    3. When you need to change your system quickly, what will enable you to do this in the fastest possible time?
    These are subjects I have written many times in my blog.But here are what I think are the 7 Key Points to remember if you want to use rule engines and business rules systems.
    P.S: I the Product Manager for QuickRules.NET, a .NET BRMS

  15. Avatar for Rajgo
    Rajgo October 4th, 2007

    Hi Phil,
    I enjoyed your post.
    Letting the business types mess directly with the system is dangerous. 100 points to you.
    business guys know what the software is supposed to do
    software guys know how it is doing it
    There are 2 issues as I see it.
    1. When the shit hits the Fan, how do the IT guys and the business types collaborate easily to resolve the issue.
    2. How do you minimize the impact of testing changes to your business rules on the rest of the system.
    3. When you need to change your system quickly, what will enable you to do this in the fastest possible time?
    These are subjects I have written many times in my blog.But here are what I think are the 7 Key Points to remember if you want to use rule engines and business rules systems.
    P.S: I the Product Manager for QuickRules.NET, a .NET BRMS

  16. Avatar for Travel Masti India
    Travel Masti India November 17th, 2010

    i got some good information from here realy good...